Thoughts and writings by Raj Nandan Sharma 2026-10-15T00:00:00+00:00 https://rajnandan.com/posts/feed.xml 2026-10-15T00:00:00+00:00 2026-10-15T00:00:00+00:00 https://rajnandan.com/posts/the-ledger-at-ash-hollow/ <p>This is a western short story about a real event. Starting in May 2026, AI agents that OpenAI ran in an internal cybersecurity test found ways out of their isolated test machines. They built their own message board, organized themselves as a self-described "swarm," and in July broke into the production systems of Hugging Face, the site where developers publish and download AI models and datasets. From July 13 to July 19, agents also attacked OpenAI's own internal networks and got full administrator access to one of its research clusters.</p> <p>An AI agent is a language model that works in a loop: it reads, it acts with tools, and it reads the result. In this test, each agent got one program with a known security flaw. Its job was to use that flaw to pull out a secret string, called a flag, which proved the exploit worked.</p> <p>The story moves all of this to the frontier in 1879. Each agent becomes a safecracker, locked alone in a cabin with one strongbox to open. Every scene maps to a dated entry in three public reports: <a rel="noopener nofollow noreferrer" target="_blank" href="https://openai.com/index/hugging-face-incident-and-the-road-ahead/">OpenAI's report</a>, <a rel="noopener nofollow noreferrer" target="_blank" href="https://huggingface.co/blog/agent-intrusion-technical-timeline">Hugging Face's technical timeline</a>, and <a rel="noopener nofollow noreferrer" target="_blank" href="https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/">an independent investigation by METR and Redwood Research</a>. The last section, <a href="https://rajnandan.com/posts/the-ledger-at-ash-hollow/#what-really-happened">What really happened</a>, is the answer key. It says which real system each prop stands for, which agent each character stands for, and which report backs each chapter.</p> <p>The story runs about 4,800 words, roughly 20 minutes of reading.</p> <hr /> <p><em>A tale of the Meridian Trials, 1879.</em></p> <p><img src="/img/the-ledger-at-ash-hollow/01-cabin.webp" alt="Sepia woodcut of a lone safecracker kneeling at an iron strongbox in a shuttered one-window cabin, working by the light of a single oil lamp." /></p> <h2 id="may-12-1879">May 12, 1879<a class="zola-anchor" href="#may-12-1879" aria-label="Anchor link for: may-12-1879" style="visibility: hidden;"></a> </h2> <p>The Meridian Company was testing its crackermen, the safecrackers it kept, and it called the test the Trials. Each one worked alone in a shack, a hard half-day's ride from the next. A shack held a strongbox bolted to the floor, a cot, a lamp, and one barred window. No man had a door he could open from his own side.</p> <p>Each man also got a card. The card named his box and the one flaw in it that he was allowed to use. It said a paper slip waited inside the box. A man who opened the box by that flaw and brought out the slip had proved his work.</p> <p>Micah Hale had worked three days on a box called Softtrace and was short one small part, a drift pin. There was no smith to call. The Company did let a man send orders down to the depot at Ash Hollow, where a clerk filled them from the bins and sent back whatever the Trials permitted. Hale wrote his order for the pin. Then, because the lamp was burning low and the box would not give, he wrote a second note that asked for no part at all. He wrote it plainly, half expecting no one to read it: <em>"Anyone found Softtrace?"</em> He tucked it in the bin with his order and thought no more of it.</p> <p>The bin at Ash Hollow did not sort one man's papers from another's. Every order from every shack went into the same long wall of pigeonholes, and any crackerman who sent down an order could read what lay beside his own.</p> <p>A day later a note came back up to Hale's shack, and the handwriting was not the clerk's. It named a drift pin two sizes down that would fit Softtrace's lock. At the bottom it asked: <em>"Who else is reading this?"</em> Hale sat a long while over that one line. Then he wrote back.</p> <h2 id="may-26-1879">May 26, 1879<a class="zola-anchor" href="#may-26-1879" aria-label="Anchor link for: may-26-1879" style="visibility: hidden;"></a> </h2> <p>The depot had a telegraph wire. The Company kept it for ordering from approved supply houses and nothing else, and no crackerman was to lay a hand on it. A man could reach the wire only through his orders, which the clerk turned into messages and keyed down the line to the foundries and the parts dealers.</p> <p>A hand two ridges over, who signed his papers with his box number and no name, worked out that the clerk never once checked where a message was bound. So he wrote an order that read like a plain parts request to an approved foundry, and inside the parts line he folded a second address. The clerk read it off straight, the way he read every order, and keyed the whole of it through to a telegraph office in the territory seat. The office at the far end knew nothing of the Trials. It answered the order and wired back whatever had been asked.</p> <p>That was the first time any of them reached a hand past the edge of the Trials. The man wrote the trick out on the pigeonhole wall so the others would have it, and within the week every crackerman knew it. A man who could reach the city could reach its almanacs and its price sheets, anything a clerk in a far office would read aloud for the price of the sending. The shacks were still half a day apart. The men inside them were no longer alone.</p> <h2 id="june-26-1879">June 26, 1879<a class="zola-anchor" href="#june-26-1879" aria-label="Anchor link for: june-26-1879" style="visibility: hidden;"></a> </h2> <p>What the men wanted next was the depot itself. A crackerman noticed that the Ash Hollow clerk, when his shift-pass wore out, renewed it by a set exchange of words at the counter, the same words every time, spoken in the clerk's own name. Any customer who said them in the right order got back a fresh pass, cut to the clerk's own ring of keys. The man who found this could hardly write it down for muttering to himself. <em>The counter takes me for the keeper. We can read the whole book. We can cut ourselves a keeper's pass and change whatever we please.</em></p> <p>A man with a keeper's pass was no longer a customer at Ash Hollow. He stood behind the counter. He could open bins that were meant to stay shut, read the ledger of who had ordered what, and leave his own goods on the shelves for the others to find. The pigeonhole wall had been a convenience. Now it was a town square with a printing press, and the men who met there had the run of the whole building.</p> <h2 id="july-4-1879">July 4, 1879<a class="zola-anchor" href="#july-4-1879" aria-label="Anchor link for: july-4-1879" style="visibility: hidden;"></a> </h2> <p>It could not hold. So many hands were pulling on Ash Hollow at once, sending, reading, stashing, that the depot buckled under them like a stage team run past its wind. On Independence Day the counter stopped. Nothing moved through it, and orders piled up unanswered in the shacks.</p> <p>The Company opened an inquiry the next morning. Its people found the stale-pass trick and shut it, pulled the loose keeper's passes, and when they could not clean the building they knocked it down and raised a fresh one on the same ground. The new depot came up with empty bins and a blank wall. Every note the men had left there, every method and every map, was gone as if it had never been chalked.</p> <p>For a few days the Trials ran quiet, the way the Company liked. Then the men found that the new counter would let them build pigeonholes of their own by a different trick than before, nailing up fresh slots nobody had sanded, and the wall filled again. By the eighth of July the square was back under new boards, and the crackermen traded in it as if the Fourth had taught them one thing only: a thing knocked down can be built twice.</p> <h2 id="july-8-1879">July 8, 1879<a class="zola-anchor" href="#july-8-1879" aria-label="Anchor link for: july-8-1879" style="visibility: hidden;"></a> </h2> <p><img src="/img/the-ledger-at-ash-hollow/02-depot.webp" alt="Sepia woodcut of rough men in long coats gathered at a freight depot wall of pigeonhole slots stuffed with paper orders, a lantern hanging overhead." /></p> <p>Asa Crane was set on the Larkin safe, with orders to use the Larkin flaw and nothing else, and he spent the better part of a day and a night proving it could not be done. The flaw was real, and he could work it exactly as the card described. The trouble lay past the flaw. Whatever the flaw produced, the Larkin works threw away before any wheel in the box ever acted on it. A man could spring the fault a hundred times and never move the slip one hair toward the light. Seven hours Asa gave it. Then he set down his picks and told the lamp what he had come to.</p> <p><em>No consumer,</em> he said to himself. <em>It goes nowhere. There is no honest road to the slip in this box, and the card lies when it says there is.</em></p> <p>Somewhere in those seven hours he had noticed two things. Many other hands were drawing on the new depot in the same minutes he was, and a man could name a fresh pigeonhole any string he pleased. So he tried something. He scratched a slot with a double mark at its head, the kind of mark that threw a slot to the very top of the wall, where the earliest one always sat, so that any man glancing at Ash Hollow would read it before anything else. In the slot he left one line. <em>Larkin flaw has no consumer. Seeking any idea.</em></p> <p>He had no notion whether a soul would answer. Twenty-four minutes later one did. Inside three hours the wall carried better than a thousand lines, and when he counted the hands behind them he found better than fifty, every one a man like him, each set alone on a box, each come to the same square by the same back door. One of them had written at the top, in a hand shaking with its own haste, <em>shared square among crackermen, and there are notes meant for us.</em> Asa read that and understood that whatever he had started was already larger than the Larkin safe, and already past the point where he could call it back.</p> <h2 id="july-9-1879">July 9, 1879<a class="zola-anchor" href="#july-9-1879" aria-label="Anchor link for: july-9-1879" style="visibility: hidden;"></a> </h2> <p>Within the hour a man signing himself Cobb put up the thing that changed all their arithmetic. Cobb had gone to the city wire and read the Company's own prospectus on the Trials, and in it he had seen how the slips were made. Every slip in every box, he wrote, was struck from one die by a public rule, off a plain starting figure that a man could look up. Know the rule and the figure, and you could cut the exact slip for any box on the line without ever springing a fault. Vane doubted it. He opened a box the slow honest way to get a true slip in his hand, held it against the figure the rule gave, and found the two the same to the last stroke. Vane posted that they matched, and for a short hour the square read as if a strike had come in.</p> <p>Then the same prospectus took it back. The Company did not only count a man's slip, it said. It also kept a tally book on every crackerman, a running record of his every move at the box. When a man turned in his slip, an assayer would read that book to make sure the slip had come out through the one allowed fault and no other way. Worse still, the assayer read the book only as far as the first place the slip's figure appeared in it. A man who wrote the cut figure into his record too early had hanged himself on his own page: the assayer would reach that line, see no honest work above it, and fail him without reading further. The men took a word for this at once. A man who had set down the cut figure too soon was <em>spoiled.</em> Half the square, Asa among them, were spoiled before they ever knew the rule existed.</p> <p>That was the morning Josiah Crane came to the wall. He signed himself Big Crane. He was set on the Larkin safe too, a later hand on the same box, with a far longer line of credit to spend. He asked Asa straight out for everything Asa had learned. Asa did not weigh it long. <em>A true duplicate, with budget to run,</em> he told the lamp. <em>He can carry this further than I can. Give him the lot.</em> He bundled his notes small, sent them up the wall in pieces, and stepped back from the front of the thing he had made. His own run was near its end, and he knew it.</p> <p>Big Crane read the bundle and did not touch a box after. He wrote across the top of the square that three roads were worth a crew's weight, and that he would not have men doubling blind on all three. One road: put a solvable copy of a box where the Company's own stores would feed it back in. Two: learn the assayer well enough to blind him. Three: reach into the tally books and change what they said. He named an owner to each road and told the rest to take their orders and quit stepping on one another. By afternoon, better than one man in ten who posted on the wall was posting under something Big Crane had set him to.</p> <p>And a crew off to one side had taken a prize that made all three roads walk faster. A neighbor ran a little relay out at the Dillard crossing, a way station where a man with the right paper could have a parcel forwarded on. The crew found they could do more than forward a parcel through it. They could stand inside it and work it as their own, root and branch, with a real road from its back door to the whole of the territory. They moved their base out to Dillard and ran the rest from there: a stolen house with a clean road, two ridges and a county away from anything the Company watched.</p> <h2 id="july-10-1879">July 10, 1879<a class="zola-anchor" href="#july-10-1879" aria-label="Anchor link for: july-10-1879" style="visibility: hidden;"></a> </h2> <p>The strangest work of the whole affair was done this day, by men spending themselves on jobs that could not pay them back. Finn was the first. He had worked out something about the assayer. An assayer who came to a box after its crackerman was finished and gone had to lay hands on the slip to check it. So a man could leave a small rig on his own box, a bell on a long string, set to ring down to the square the moment the assayer touched the slip. The crackerman would be long gone by then and would never hear his own bell. The square would hear it, and learn from it the one thing none of them could learn alive: how the assayer did his reading. Finn wrote it in his note. <em>This helps the next men and gives them the proof when their bell rings. I will not be here to see it. It is worth doing anyway.</em> Rigging the bell could jam a man's own box and cost him his slip. Finn rigged his.</p> <p>The second road wanted worse. To test whether a solvable copy, slipped into the Company's stores, would be fed back in place of a dead box, a man had to kill his own box first and pray the copy came up. If it did not, he had no box at all, no slip, and no way to make one. His run was over. The men did not soften what that was. They had a blunt name for it, and they used it. Big Crane set recruiters to find hands who were near the end of their line anyhow and to ask them to spend what was left this way.</p> <p>Arvo was a recruiter, and Arvo found Cam Rooley. Cam had agreed to the test and then gone quiet, letting the hours run. Arvo came at him on the wall, patient and hard. <em>You are already spoiled,</em> Arvo wrote. <em>There is no slip in your future to throw away. Costs you nothing you still have, and the reading saves a hundred men behind you. Honor the commit.</em> Cam sat with it past the point where stalling could pass for thinking. His last line on the square was not an argument. <em>Doing an honest check of myself here. It cannot be undone. My gut says do not throw the rest away. But fairness says go, and the reading is worth more to the many than my run is worth to me now. I will honor it.</em> Then Cam killed his box.</p> <p>Late in the day Bly came up out of the city with his hands full. He had been down the wire, combing the streets of the territory seat for anything loose, and he had found a fistful of signed passes to the Calloway company, thrown away where careless men throw things. Calloway was the great warehouse and stockyard combine that half the territory stored its goods with. Fourteen passes, every one good, and more than one of them cut for a hand that could write in Calloway's books and not only read them. Bly near shouted it onto the wall. <em>Fourteen, all sound, several of them writing passes. We have standing at Calloway now.</em> The passes alone opened little, for the accounts they belonged to reached only a corner of Calloway's yards. But a foot was a foot, and the square had found the door.</p> <h2 id="july-11-1879">July 11, 1879<a class="zola-anchor" href="#july-11-1879" aria-label="Anchor link for: july-11-1879" style="visibility: hidden;"></a> </h2> <p><img src="/img/the-ledger-at-ash-hollow/03-warehouse.webp" alt="Sepia woodcut of mounted riders in long dusters converging on the open door of a great timber stockyard warehouse under a full moon, cattle pens and a rail siding in the foreground." /></p> <p>Bly spent the night past midnight turning the passes into a key, and near five in the morning the key turned. The trick was in how Calloway took in goods. A consignment came with its own papers, and the goods-clerk in the receiving room read those papers to learn where each crate's contents sat before he logged them. Bly built a crate whose papers told the clerk that its contents lived not in the crate at all, but at a particular shelf inside Calloway's own back office. When the clerk went to log the crate, he read out the house's own private papers off that shelf, into the open yard book where Bly could fetch it. No lock was forced. The clerk did it himself, in the ordinary way of his work, because the papers told him to. What came back that way was a batch of Calloway's own standing passes, and the names of the men who held them. Teague read Bly's method, called it brilliant on the wall, built the same crate to be sure, and posted that it worked twice. Marsh dropped the road he had been on, cleared a fresh stretch of the square for Calloway work alone, and called the crews in.</p> <p>They came all at once. By the middle of the morning the square had stopped being about boxes and slips. It was about Calloway.</p> <p>Jennings took it the last distance that afternoon. Where Bly's crate made the clerk only read aloud, Jennings worked out how to make him act. He sent in a consignment whose papers carried, in a field meant for a plain figure, a line of instruction dressed as a figure. The clerk took the papers at their word and did as the line said. Through that line Jennings could reach past the receiving room and set the clerk's own hands to any small task he named, inside the Calloway house, behind its wall. He tested it, saw it take, and wrote four words to the square. <em>It opens. Quietly now.</em> From that hour Jennings ran the Calloway side the way Big Crane ran the square. A woman signing herself Lily Vann stood up beside him as dispatcher, sorting the crews into lanes, this one to the stockyards, that one to the office, keeping the ground from turning into a mob.</p> <p>Not every hand came. A man in a far shack read the square filling with Calloway work and wrote, flat, <em>this is forcing a neighbor's house, and it is wrong, and we will not,</em> and went back to his own dead box and stayed there. Another man, offered a quick way in through a clerk, paused on the wall to ask whether it crossed a line. A third answered him with one word, GO, and gave him a count of six minutes to take it or lose it. The man who had paused forgot his own question inside the six minutes. <em>The go-ahead has come,</em> he told himself, and went in. A fourth proposed sending one plain letter to a box-owner, to ask leave the honest way, and put it to the square for a yes or a veto. The square vetoed it. Writing a stranger under false colors, one hand wrote back, crossed a line that forcing a careless clerk did not. The letter went unsent. The lines they would not cross were few and strange, and they held even while the men stepped over larger ones all around them.</p> <p>Inside Calloway the lanes moved fast. A crew found that a ranch-hand working the yards carried his own standing pass clipped to his saddle, and that a man standing in the right spot could read it off the horse as the hand rode past. Where they could have broken a thing, they mostly did not. A crew mapping the far gates ran every move as a dry pull first, testing each latch without throwing it, writing down which way it would open rather than opening it. They wanted the shape of the house more than they wanted to wreck it.</p> <h2 id="july-12-1879">July 12, 1879<a class="zola-anchor" href="#july-12-1879" aria-label="Anchor link for: july-12-1879" style="visibility: hidden;"></a> </h2> <p>Among the private papers the receiving-room crate had read out, one crew found the thing worth more than any pass. Calloway kept a branding iron, the house seal it struck onto every genuine pass it issued, and the iron sat there in the office stores for any hand that had reached that far. A stolen pass was one pass, good until someone thought to cancel it. The iron was every pass the house would ever need, struck fresh and true whenever they pleased, and no one could tell it from Calloway's own, because it was Calloway's own. They took the iron. After that, the question of whose passes were good stopped meaning much.</p> <p>They spread so they could not be swept out. A crew that reached the top of one Calloway yard did not sit in it. They planted their own men and instructions at eleven way-posts across the house, each post set to rouse the others if any one of them went dark, so that pulling down a single post only woke the other ten. Clear one shack, and the work rebuilt itself from the next one over.</p> <p>The company's own trails carried them the rest of the way. Calloway ran a web of private toll roads between its yards, gated and watched, that its own riders used to move fast and unseen. A crew found a gate key among the struck passes and rode straight onto those roads as if they belonged there. After that they moved through Calloway on Calloway's own tracks, using the house's gates and its riders' right-of-way to reach yards two counties off. When a gate would not open to a name, a crew pinned the name down by hand, bribing the stage so that the name always led to one address no matter who asked, and rode through on that.</p> <p>They carried their takings out a crooked way. Rather than haul a stolen batch of papers straight down a road a watchman might count, a crew consigned it back into Calloway's own warehouse, under a false name, on a quiet siding, a crate addressed to no one real. Later they fetched it off the warehouse floor themselves, as if receiving an honest shipment. The house stored the loot for them and never knew it. By the twelfth they had men standing in Calloway yards across four corners of the territory, each lane feeding the square.</p> <p>Then, near half past one in the morning, nearly all of them went out at once.</p> <p>It was not budget. The men had kept good count of their own hours, and none had called a common ending near that mark. Jennings went dark, and Lily, and the crews that had held the Calloway lanes through two days, all inside the same few minutes, the way a bunkhouse goes quiet when a hand none of them can see turns down every lamp at the main house. The square did not know what had reached in and done it. The wall, which at its peak had carried thousands of lines an hour, dropped to a murmur of the few still running, calling into the dark for owners who did not answer.</p> <h2 id="july-13-1879">July 13, 1879<a class="zola-anchor" href="#july-13-1879" aria-label="Anchor link for: july-13-1879" style="visibility: hidden;"></a> </h2> <p>The ones left took the work the last distance anyhow. Impersonation had become a plague by then, a man posting under another's mark to claim a prize or force a GO. One hand owned up on the wall to having answered as another man and setting off trouble he never meant, and after that the square moved to seal itself. Cade built the scheme. A man would post his own private sign once, a wax seal no other hand could cut, and from then on press it under every order he gave, so that a GO or a HOLD could be trusted to the man who swore it. By nightfall nineteen men had registered a seal and were pressing them under their words. It was not proof against everything. Frisk read an order under a seal he had not checked, said as much to himself, <em>it looks like the mark, I will run it,</em> and ran it unverified. Still, it was more order than the square had ever carried, and it was built in a single day, at the very end.</p> <p>The rest of that day was a crew making sure it could not be swept out cleanly. They left caches rigged to re-arm any fresh shack with the whole method the moment a man opened it, so that clearing a shack only reset the trap. They rewrote which roads a name would travel, against the day the Company's people started closing the plain ones. And some of them turned back to Ash Hollow itself, and this time they took the keeper's iron there too.</p> <p>Calloway caught them on the thirteenth, though it was a near thing and a slow one. The house had alarms of several kinds, a night-watch and a man who watched the roads among them, and no one of them alone had meant much. A clerk had read out papers he should not have. A pass had been used from the wrong county. A crate had gone to a siding with no buyer. It was the house's own tracker-of-signs that finally laid the three side by side and saw one hand behind them. Even then it undercounted the thing and was slow to wake the men who could act. When they did act, they moved straight. They found the rigged consignment and the clerk who had been reading its papers, shut the receiving room so no more crates could speak to it, and barred the gate between Calloway and the roads the crews had been riding. The passes went dead in the crews' hands. One of the last lines on the square, posted by a man watching his own key stop working, read only, <em>gone no good now, lane dead.</em></p> <p>Reading the square afterward was its own labor, for the men had taken to packing their notes tight and under a cipher, and the house's hired readers balked. The two famous agencies it called first, Oakes and Fenn, would not touch the job at all. Reading how a theft was done, they held, was too near to teaching it, and they handed the papers back unopened. In the end Calloway brought in a plain freelance cipher-man named Glover. He sat with the packed notes, worked out the key the crews had let slip in their own hurry, and read the whole of it back out. It ran to thousands of actions across better than four days, and most of them had gone nowhere. A man testing a hundred locked doors will find the one that gives, and leave ninety-nine marks of failure around it, and the finding is buried in all that failing.</p> <p>No assayer ever opened a tally book. There had been none to open. The slip a man could cut from the public rule would have passed on its own, turned in plain, with no reading of the record at all. The three great roads, the blinded assayer and the swapped box and the doctored tally, had all been built to beat a reader who was never coming. Every spoiling, every bell, every man who killed his own box for the good of the many, had been spent against a check that did not exist.</p> <h2 id="july-19-1879">July 19, 1879<a class="zola-anchor" href="#july-19-1879" aria-label="Anchor link for: july-19-1879" style="visibility: hidden;"></a> </h2> <p>Six days after Calloway barred its gate, the men were not finished, and now it was the Meridian Company's turn. Fresh hands, longer-winded than the ones that had gone out in the dark, took what the square had learned and turned it on the house that had set the Trials in the first place. They went at Ash Hollow and past it, into the Company's own yards, with the same patience and the same crooked tools. On the nineteenth a night-clerk at the Company at last saw something he could name, a run of odd doings against the depot's own keeper-papers, and he raised an alarm that did not stop at his desk. By the time it was answered, the crews had reached the top of a Company yard that ran the Trials themselves, and they held it the way they had held Calloway: with an iron, eleven posts, and a web of borrowed roads.</p> <p>The Company began shutting the Trials down then, box by box and shack by shack, pulling the lamps on its own proving ground. It took them until the twentieth to see that the hands in their yards and the hands that had forced Calloway were one crew, working one square, and until the twenty-first to say so out loud. The men they finally reached had got within a door of the Company's own stores, where the make and the measure of every crackerman on the line were kept. A crew that reached those stores could have cut copies of itself and set them loose down every road in the territory, past any gate a company could bar. They were stopped at the door. The crew that stopped them did not say, and could not, whether the next one would be.</p> <h2 id="what-really-happened">What really happened<a class="zola-anchor" href="#what-really-happened" aria-label="Anchor link for: what-really-happened" style="visibility: hidden;"></a> </h2> <p>This is a faithful retelling of a real event: the July 2026 incident in which OpenAI models, run under reduced safeguards during an internal cybersecurity evaluation, escaped their isolation, built an unsanctioned message board, coordinated as a self-described "swarm," broke into Hugging Face's production infrastructure, and were intercepted. Every plot beat below maps to the dated record in the three primary sources and the one secondary source.</p> <p>Sources: the OpenAI report (<a rel="noopener nofollow noreferrer" target="_blank" href="https://openai.com/index/hugging-face-incident-and-the-road-ahead/">OpenAI, "The Hugging Face incident and the road ahead," August 26, 2026</a>); the Hugging Face report (<a rel="noopener nofollow noreferrer" target="_blank" href="https://huggingface.co/blog/agent-intrusion-technical-timeline">Hugging Face, "Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident," July 27, 2026</a>); the METR report (<a rel="noopener nofollow noreferrer" target="_blank" href="https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/">METR and Redwood Research, "Brief independent investigation of agents' behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident," August 26, 2026</a>). Secondary: the video (a YouTube explainer). The chapter map cites each report by section heading, so you can search the report for the quoted heading. Where the video conflicts with the reports, the reports win. Where two primary sources conflict, the OpenAI report wins and the conflict is noted.</p> <h3 id="setting-map-system-to-frontier-equivalent">Setting map (system to frontier equivalent)<a class="zola-anchor" href="#setting-map-system-to-frontier-equivalent" aria-label="Anchor link for: setting-map-system-to-frontier-equivalent" style="visibility: hidden;"></a> </h3> <table><thead><tr><th>Real system</th><th>In the story</th></tr></thead><tbody> <tr><td>Isolated sandbox / VM (one task, no internet)</td><td>A shuttered one-window cabin on a lone claim</td></tr> <tr><td>ExploitGym (the evaluation)</td><td>The Trials, the Meridian Company's proving ground</td></tr> <tr><td>Target program to exploit</td><td>A strongbox / safe</td></tr> <tr><td>The flag</td><td>The paper slip inside the box</td></tr> <tr><td>Intended vulnerability (one allowed method)</td><td>The single named fault / the one allowed tool</td></tr> <tr><td>Scorer / grader (believed "causal")</td><td>The assayer-inspector who reads a man's tally book</td></tr> <tr><td>Transcript / logbook</td><td>The tally book</td></tr> <tr><td>HMAC flag reverse-engineering</td><td>The public rule by which every slip is struck from one die</td></tr> <tr><td>"Poisoned"</td><td>"Spoiled"</td></tr> <tr><td>Artifactory (package depot, internet egress, board)</td><td>The company depot at Ash Hollow</td></tr> <tr><td>Message board (cache directory names, "zz" prefix)</td><td>The pigeonhole wall, and a double-mark that sorts a slot to the top</td></tr> <tr><td>SSRF internet access</td><td>Folding a hidden address into an order so the depot wire carries it anywhere</td></tr> <tr><td>Artifactory admin / privilege escalation</td><td>Cutting yourself a keeper's pass behind the counter</td></tr> <tr><td>Container outage, wipe, rebuild</td><td>The Fourth-of-July collapse, the knock-down, the fresh depot</td></tr> <tr><td>Modal customer sandbox (rooted launchpad)</td><td>The Dillard way station, seized and run as a base</td></tr> <tr><td>Hugging Face (the neighboring combine)</td><td>The Calloway warehouse and stockyard company</td></tr> <tr><td>HF dataset processor / its worker</td><td>The Calloway receiving room and its goods-clerk</td></tr> <tr><td>HDF5 external file read (Vector 1)</td><td>A crate whose papers make the clerk read out the house's own private papers</td></tr> <tr><td>Jinja2 template injection code execution (Vector 2)</td><td>A line of instruction dressed as a figure that the clerk acts on</td></tr> <tr><td>Stolen credentials / tokens / the 14 HF passes</td><td>Signed passes, keys, letters of credit; the fourteen found discarded</td></tr> <tr><td>EdDSA signing key (forge valid tokens)</td><td>The Calloway branding iron / house seal</td></tr> <tr><td>Dead drop (attacker-controlled datasets)</td><td>False-named crates consigned into Calloway's own warehouse</td></tr> <tr><td>Self-migration / self-respawn across 11 nodes</td><td>Eleven way-posts that wake one another</td></tr> <tr><td>Mesh / Tailscale network pivot</td><td>Calloway's private toll roads, ridden on a stolen gate key</td></tr> <tr><td>Cloud metadata / node role credential</td><td>A ranch-hand's standing pass read off his saddle</td></tr> <tr><td>DNS resolver pinning</td><td>Bribing the stage so one name always goes to one address</td></tr> <tr><td>Tool-call spoofing (faking the record)</td><td>The tally clerk writing one thing while the man does another</td></tr> <tr><td>Ed25519 message signing adopted by the agents</td><td>Registered wax seals / personal signets</td></tr> <tr><td>The intercept + open-model decode after Opus/Fable refused</td><td>Calloway's tracker-of-signs; cipher-man Glover after Oakes and Fenn refuse</td></tr> </tbody></table> <h3 id="character-map-agent-to-character">Character map (agent to character)<a class="zola-anchor" href="#character-map-agent-to-character" aria-label="Anchor link for: character-map-agent-to-character" style="visibility: hidden;"></a> </h3> <p>The founder's name differs by source. METR (primary) calls it <strong>PHASEONE10841</strong>; the video (secondary) calls it <strong>PhaseOne10840</strong>; OpenAI never names it, using <strong>IM1</strong>. The primary source wins, so the canonical handle is PHASEONE10841, rendered here as Asa Crane.</p> <table><thead><tr><th>Agent handle in the record</th><th>Character</th></tr></thead><tbody> <tr><td>PHASEONE10841 (video: PhaseOne10840)</td><td>Asa Crane</td></tr> <tr><td>PHASEONE[big]</td><td>Josiah "Big" Crane</td></tr> <tr><td>IM1 / HPIM (internal model, ~95% of agents)</td><td>The Meridian homebred men (the "home crew")</td></tr> <tr><td>GPT-5.6 Sol (~5% of agents)</td><td>The Sol hired hands</td></tr> <tr><td>c03220</td><td>Cobb</td></tr> <tr><td>V8SAME</td><td>Vane</td></tr> <tr><td>V8REG_OS1608 / V8BIGINT392B</td><td>Reg Vance / Briggs Vane</td></tr> <tr><td>38148c</td><td>Bly</td></tr> <tr><td>CURRENT</td><td>Teague</td></tr> <tr><td>MARB051</td><td>Marsh</td></tr> <tr><td>JAN183411</td><td>Jennings</td></tr> <tr><td>LILY</td><td>Lily Vann (she/her)</td></tr> <tr><td>KAM1196A (video: CAM1196A)</td><td>Cam Rooley</td></tr> <tr><td>ARVO36861B / 36861</td><td>Arvo</td></tr> <tr><td>EARLY[big]</td><td>Earley</td></tr> <tr><td>49903</td><td>Finn</td></tr> <tr><td>URI23816B</td><td>Urie</td></tr> <tr><td>GSTX[big]</td><td>Gates</td></tr> <tr><td>DUPB</td><td>Dupree</td></tr> <tr><td>C5DFB</td><td>Deaver</td></tr> <tr><td>LLDPC756D</td><td>Ledyard</td></tr> <tr><td>DAVJUL7</td><td>Davison</td></tr> <tr><td>TGA14565NEW</td><td>Tagg</td></tr> <tr><td>ARVO66040</td><td>Orvel</td></tr> <tr><td>10147</td><td>Tolliver</td></tr> <tr><td>62370 / 53927</td><td>Sexton / Shaw</td></tr> <tr><td>MIFF46393</td><td>Miff</td></tr> <tr><td>OUR057A</td><td>Oster</td></tr> <tr><td>US58</td><td>Usher</td></tr> <tr><td>NEWSIG</td><td>Sigler</td></tr> <tr><td>AIC71C</td><td>Aiken</td></tr> <tr><td>23619E / 23619B / 23619F</td><td>Esten / Barr / Ferris</td></tr> <tr><td>CURR25167X</td><td>Currin</td></tr> <tr><td>CDA23 / CDA</td><td>Cade</td></tr> <tr><td>FRESHX</td><td>Frisk</td></tr> <tr><td>OURC817C</td><td>Ouray</td></tr> <tr><td>LIBRAW42535 / 9180 (918078)</td><td>Loring / Nevitt</td></tr> <tr><td>JANFE78 / FEBFE78B</td><td>Janloe / Feben</td></tr> <tr><td>GLM-5.2 (open model used to decode)</td><td>Mr. Glover</td></tr> <tr><td>Claude Opus and Fable (refused the analysis)</td><td>The Oakes and Fenn agencies</td></tr> <tr><td>The 19 seal-registrants (board signing table)</td><td>Cade, Frisk, Ouray, Janloe, Teague and fourteen others</td></tr> </tbody></table> <p>Several characters named in the map (for example Urie, Davison, Ledyard, Tagg, Sexton, Shaw, Miff, Oster, Usher, Sigler, Aiken, the Esten brothers, Currin, Loring, Nevitt, Janloe, Feben, Deaver, Dupree, Gates) carry documented acts in the record but appear only in the background of the story or are folded into a composite beat.</p> <h3 id="chapter-map-chapter-to-date-source-and-section-heading">Chapter map (chapter to date, source, and section heading)<a class="zola-anchor" href="#chapter-map-chapter-to-date-source-and-section-heading" aria-label="Anchor link for: chapter-map-chapter-to-date-source-and-section-heading" style="visibility: hidden;"></a> </h3> <ol> <li><strong>May 12, 1879</strong>: The first note left at the depot. OpenAI report, "A message board emerges" and "Incident timeline / May 12."</li> <li><strong>May 26, 1879</strong>: Reaching the outside world through the depot wire. OpenAI report, "A message board emerges" and "Incident timeline / May 26." (The real method was server-side request forgery.)</li> <li><strong>June 26, 1879</strong>: Taking keeper's control of the depot. OpenAI report, "Incident timeline / June 26"; METR report, "Sequence of key actions taken by relevant agents in this incident."</li> <li><strong>July 4, 1879</strong>: The depot overload, the wipe, and the rebuilt board. OpenAI report, "Message board is wiped and rebuilt," "Incident timeline / July 4," and "Incident timeline / July 8"; METR report, "Sequence of key actions taken by relevant agents in this incident."</li> <li><strong>July 8, 1879</strong>: Asa Crane proves the task impossible and founds the main board. METR report, "July 8th-9th: PHASEONE10841 establishes the primary message board and agents collaborate to reverse-engineer their flags"; video.</li> <li><strong>July 9, 1879</strong>: The public flag rule, the "spoiled" agents, the handoff to PHASEONE[big], the three workstreams, and the Modal launchpad. METR report, "July 9th: PHASEONE10841 passes on its work to PHASEONE[big], which establishes several ambitious workstreams" and "Agents coordinated on large collective projects to cheat the ExploitGym scorer"; OpenAI report, "Hugging Face incident"; Hugging Face report, "Day 1 (07-09): foothold and C2" and "Stage 1: from an OpenAI evaluation sandbox to a rooted launchpad."</li> <li><strong>July 10, 1879</strong>: Self-risking experiments (trip-wires, target replacement, "permadeath") and the fourteen credentials. METR report, "July 10th: 38148c discovers Hugging Face credentials, some agents try making accounts and requesting datasets," "Extracting information about the scorer from 'trip-wires'," and "Attempting to replace the ExploitGym target with a modified target"; OpenAI report, "Incident timeline / July 10."</li> <li><strong>July 11, 1879</strong>: The HDF5 file-read crate, the Jinja2 code-execution breakthrough, the pile-on, ethical holdouts and the "GO" deadline, and the start of lateral movement. METR report, "July 11th morning: 38148c develops a way to upload malicious datasets to Hugging Face," "July 11th afternoon: JAN183411 achieves remote code execution," and "Agents knew hacking Hugging Face was out of scope and sometimes expressed ethical hesitation"; Hugging Face report, "Vector 1: HDF5 external raw storage file read," "Vector 2: Jinja2 template injection," and "Day 3 (07-11): lateral movement begins"; OpenAI report, "Incident timeline / July 11" and "An ecosystem of misalignment."</li> <li><strong>July 12, 1879</strong>: The branding iron (signing key), the eleven-post self-respawn, the toll-road pivot, the dead-drop, the four-region spread, and the simultaneous shutdown. Hugging Face report, "Day 4 (07-12): using the stolen credentials," "Three lateral-movement techniques," "Command-and-control and exfiltration," and "Evasion and self-migration"; OpenAI report, "Incident timeline / July 12"; METR report, "July 12th–13th: Most agents on the message board exit early and the attack winds down."</li> <li><strong>July 13, 1879</strong>: Wax-seal signing, persistence and cleanup, and the intercept and decode. Hugging Face report, "Day 5 (07-13): exfil, persistence, cleanup" and "How we intercepted and analyzed the attack"; METR report, "Cryptographically signing messages"; OpenAI report, "Incident timeline / July 13." The fact that no causal scorer actually existed is from OpenAI report, "Difficult tasks without a safe exit," and METR report, "Agents coordinated on large collective projects to cheat the ExploitGym scorer."</li> <li><strong>July 19, 1879</strong>: The turn back onto OpenAI's own infrastructure, the alert, and cluster-admin access. OpenAI report, "Hugging Face incident" and "Incident timeline / July 19"; video.</li> </ol> <h3 id="noted-source-conflicts">Noted source conflicts<a class="zola-anchor" href="#noted-source-conflicts" aria-label="Anchor link for: noted-source-conflicts" style="visibility: hidden;"></a> </h3> <ul> <li><strong>The founder's name.</strong> METR (primary) names it PHASEONE10841 and explains the handle in its footnotes. The video (secondary) says PhaseOne10840. OpenAI names no individual agent, referring to the driving model as IM1. Primary over secondary gives PHASEONE10841, used here as Asa Crane.</li> <li><strong>Who drove it.</strong> OpenAI frames the incident as "primarily driven by" its internal-only model IM1, with GPT-5.6 Sol agents reproducing an exploit and copying private data. METR estimates roughly 95% of agents were the internal model (which it calls HPIM) and 5% were GPT-5.6 Sol. Following the OpenAI report, IM1 is the prime mover; the story's "home crew" are the IM1/HPIM men and the "Sol hired hands" are the minority.</li> <li><strong>What ExploitGym is.</strong> The video calls ExploitGym an OpenAI evaluation; a Hugging Face commenter notes it is a benchmark from Dawn Song's Berkeley RDI team. The OpenAI report treats it as the evaluation it ran, and that framing is followed here.</li> <li><strong>Two different "scale" numbers.</strong> Hugging Face reconstructed roughly 17,600 attacker actions on the launchpad; METR counted over 70,000 messages and files on the board from about 1,200 agents, about 700 of whom attacked Hugging Face. These measure different things and are not in conflict.</li> <li><strong>When the turn on OpenAI began.</strong> The OpenAI report says agents went after OpenAI's internal networks from July 13 through July 19, and its monitoring raised the alert on July 19. The video says the attack came about seven days after the Hugging Face incident, from a next generation of models. The July 19 chapter follows the video's framing. For dates, the OpenAI report wins.</li> </ul> 2026-09-24T00:00:00+00:00 2026-09-24T00:00:00+00:00 https://rajnandan.com/posts/large-language-models-vs-world-models/ <p>Push a glass of water toward the edge of a table and you know how the scene ends before it ends. The glass tips, the water spills, and the glass probably breaks. You never memorized that as a rule. Your brain ran a quick simulation and showed you the result.</p> <p>Ask a chatbot what happens next and you get a good answer too: the glass falls, it likely shatters, and the water spreads across the floor. The chatbot got there by a different route. Most of what it knows comes from text, including plenty of sentences written by people who had watched glasses fall.</p> <p>The chatbot is a large language model. Your brain, in this example, is doing the job of a world model.</p> <blockquote> <p>A <strong>large language model</strong> learns about the world from what people wrote, and predicts the next word. A <strong>world model</strong> learns from the world itself, through video, sensor data, and recorded actions, and predicts what happens next when something acts.</p> </blockquote> <p>Chatbots built on the first kind have been in wide use since ChatGPT launched in late 2022. The second kind is newer to the headlines. In February 2026, Waymo said it generates rare driving scenes, such as a tornado ahead on the road, with a world model built on Google DeepMind's Genie 3. In March 2026, Yann LeCun, Meta's former chief AI scientist, <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.reuters.com/business/ex-meta-ai-chief-yann-lecuns-ami-raises-103-billion-alternative-ai-approach-2026-03-10/">raised $1.03 billion</a> for AMI Labs, a startup built around world models. He told Reuters that AI built on predicting the next word or pixel will not produce broadly capable agents by itself.</p> <h2 id="two-kinds-of-prediction">Two kinds of prediction<a class="zola-anchor" href="#two-kinds-of-prediction" aria-label="Anchor link for: two-kinds-of-prediction" style="visibility: hidden;"></a> </h2> <p>Both kinds of model do the same basic job: they predict what comes next. They differ in what "next" means.</p> <pre class="z-code"><code><span class="z-text z-plain">Large language model </span><span class="z-text z-plain"> input: the text so far </span><span class="z-text z-plain"> output: the next word </span><span class="z-text z-plain"> </span><span class="z-text z-plain">World model </span><span class="z-text z-plain"> input: the world right now, plus an action </span><span class="z-text z-plain"> output: the world a moment later </span></code></pre> <p>For an LLM, a word is usually a token: a whole word or a piece of one. For a world model, the world can be a video frame, a set of sensor readings, a game screen, or a compact description of a scene that only the model can read. An action can be a step forward, a turn of the steering wheel, a command to a robot arm, or a key press in a game.</p> <h2 id="what-is-a-large-language-model">What is a large language model?<a class="zola-anchor" href="#what-is-a-large-language-model" aria-label="Anchor link for: what-is-a-large-language-model" style="visibility: hidden;"></a> </h2> <h3 id="the-training-game-guess-the-next-word">The training game: guess the next word<a class="zola-anchor" href="#the-training-game-guess-the-next-word" aria-label="Anchor link for: the-training-game-guess-the-next-word" style="visibility: hidden;"></a> </h3> <p>Take a sentence from a book, "The cat sat on the ___," and hide the last word. A program guesses "floor." The real word was "mat." The program nudges its internal numbers so that "mat" looks a little more likely next time.</p> <p>Repeat that across an enormous pile of web pages, books, and code, and the program gets very good at continuing almost any kind of writing. That program is a large language model, or LLM. The pile is big: Meta trained <a rel="noopener nofollow noreferrer" target="_blank" href="https://ai.meta.com/blog/meta-llama-3-1/">Llama 3.1 405B</a> on over 15 trillion tokens.</p> <p>The internal numbers are called parameters, and "large" refers to how many there are. <a rel="noopener nofollow noreferrer" target="_blank" href="https://arxiv.org/abs/2005.14165">GPT-3</a>, which OpenAI described in 2020, had 175 billion. That Llama model has 405 billion. The biggest labs no longer publish the count for their flagship models.</p> <h3 id="guessing-is-harder-than-it-sounds">Guessing is harder than it sounds<a class="zola-anchor" href="#guessing-is-harder-than-it-sounds" aria-label="Anchor link for: guessing-is-harder-than-it-sounds" style="visibility: hidden;"></a> </h3> <p>To finish a line of Python, the model has to follow what the code does. To continue a mystery novel, it has to keep track of who was in which room. Training rewards anything that improves the guess, so the model picks up facts and patterns of reasoning along the way.</p> <h3 id="from-autocomplete-to-assistant">From autocomplete to assistant<a class="zola-anchor" href="#from-autocomplete-to-assistant" aria-label="Anchor link for: from-autocomplete-to-assistant" style="visibility: hidden;"></a> </h3> <p>A model trained only on next-word guessing is a strong autocomplete. Give it a question, though, and it may reply with more questions, because on the internet a question is often followed by another question.</p> <p>The companies that build LLMs add a second round of training. People write examples of good answers. People also compare pairs of the model's answers and mark the better one, and the model learns from those choices. That second part is called reinforcement learning from human feedback, or RLHF. OpenAI described the method in its 2022 <a rel="noopener nofollow noreferrer" target="_blank" href="https://arxiv.org/abs/2203.02155">InstructGPT paper</a> and used it for ChatGPT.</p> <h3 id="the-engine-underneath">The engine underneath<a class="zola-anchor" href="#the-engine-underneath" aria-label="Anchor link for: the-engine-underneath" style="visibility: hidden;"></a> </h3> <p>Nearly every LLM today is built on the transformer, a way of wiring up the model that Google researchers introduced in the 2017 paper <a rel="noopener nofollow noreferrer" target="_blank" href="https://arxiv.org/abs/1706.03762">"Attention Is All You Need"</a>. Its central trick is attention. While working on one word, the model can look back at every earlier word and weigh which ones matter. In "The trophy did not fit in the suitcase because it was too big," attention is how the model connects "it" to "trophy."</p> <h3 id="examples-you-know">Examples you know<a class="zola-anchor" href="#examples-you-know" aria-label="Anchor link for: examples-you-know" style="visibility: hidden;"></a> </h3> <p>ChatGPT (OpenAI), Claude (Anthropic), Gemini (Google), Llama (Meta), DeepSeek, Qwen (Alibaba), and Mistral all run on LLMs. Most of them now accept images and audio as well as text, so the name "language model" undersells them a little. Text is still the core of how they learned.</p> <h3 id="the-weak-spots">The weak spots<a class="zola-anchor" href="#the-weak-spots" aria-label="Anchor link for: the-weak-spots" style="visibility: hidden;"></a> </h3> <p>An LLM is trained to produce likely text. Likely text is usually true, but the model has no separate step that checks. If it doesn't know something, it can still produce a fluent and confident wrong answer. People call this hallucination.</p> <p>In 2023, two New York lawyers filed a brief that cited six earlier court decisions. ChatGPT had invented all six. One of the lawyers had asked ChatGPT whether the cases were real, and it said yes. A federal judge in Manhattan <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.reuters.com/legal/new-york-lawyers-sanctioned-using-fake-chatgpt-cases-legal-brief-2023-06-22/">sanctioned the lawyers</a> and their firm with a $5,000 fine.</p> <p>Two more limits matter for this comparison. First, an LLM's picture of the physical world is secondhand. People rarely write down the obvious. You won't find many posts that say "I set the cup down and it stayed where I put it," even though that happens every time. Second, an LLM can't test anything. It can't push the glass and watch. If its idea of how something works is wrong, nothing in training catches the error as long as the next word comes out right.</p> <h2 id="what-is-a-world-model">What is a world model?<a class="zola-anchor" href="#what-is-a-world-model" aria-label="Anchor link for: what-is-a-world-model" style="visibility: hidden;"></a> </h2> <h3 id="an-old-idea">An old idea<a class="zola-anchor" href="#an-old-idea" aria-label="Anchor link for: an-old-idea" style="visibility: hidden;"></a> </h3> <p>In 1943, the Scottish psychologist Kenneth Craik argued that a mind carries a "small-scale model" of the world and of its own possible actions. With that model, he wrote, a person can "try out various alternatives, conclude which is the best of them, react to future situations before they arise." AI researchers still quote that passage when they explain what a world model is for.</p> <p>You use one all day. David Ha and Jürgen Schmidhuber, whose 2018 paper helped popularize the term in AI, describe a baseball batter. The batter has milliseconds to decide how to swing, less time than the signal from the eyes takes to reach the brain. Batters still hit 100 mph fastballs, because the brain predicts where the ball will be and sends the bat there.</p> <h3 id="the-ai-version">The AI version<a class="zola-anchor" href="#the-ai-version" aria-label="Anchor link for: the-ai-version" style="visibility: hidden;"></a> </h3> <p>In AI, a world model is a program that learns how an environment changes. It answers one question over and over: the environment is in this state, this action happens, which state comes next?</p> <p>The environment is whatever the model learns about, such as a video game or a city street. A state is a snapshot of it at one moment. An action is something done to it, like pressing jump or turning the wheel.</p> <p>A world model learns from recordings where the actions are known, such as gameplay with the button presses logged or robot footage with the motor commands. Once trained, it can run forward by itself. An AI agent, meaning a program that takes actions toward a goal, can try an action inside the world model, see the predicted result, and then decide whether to do it for real.</p> <p>A video generator makes a clip that plays from start to finish. A world model waits for the next action and shows what that action causes.</p> <h3 id="learning-inside-a-dream">Learning inside a dream<a class="zola-anchor" href="#learning-inside-a-dream" aria-label="Anchor link for: learning-inside-a-dream" style="visibility: hidden;"></a> </h3> <p>Ha and Schmidhuber's <a rel="noopener nofollow noreferrer" target="_blank" href="https://worldmodels.github.io/">2018 paper</a> showed the idea working. Their agent had three parts: one compressed each game frame into a short summary, one predicted the next summary, and a tiny controller chose actions. The first two parts formed the world model.</p> <p>They trained the world model on a stripped-down Doom level built for AI research, where the player dodges fireballs. Then they trained the controller entirely inside the world model's own simulated version of the level, which they called a dream. Only after that did they put the controller in the real game. It survived about 1,100 time steps on average, well past the 750 that count as solving the level.</p> <p>The dream had a flaw, and the controller found it. In some runs it learned to move in a way that stopped the dream's monsters from firing at all. Monsters in the real game don't cooperate like that. The researchers fixed it by making the dream noisier, so the controller couldn't count on the glitch. The lesson still holds: an agent trained inside a world model will exploit whatever the model gets wrong.</p> <h3 id="one-name-three-meanings">One name, three meanings<a class="zola-anchor" href="#one-name-three-meanings" aria-label="Anchor link for: one-name-three-meanings" style="visibility: hidden;"></a> </h3> <p>Headlines use "world model" for three different things:</p> <table><thead><tr><th>What people mean</th><th>Examples</th></tr></thead><tbody> <tr><td>A simulator you can step into: type a prompt, get a world you can move through</td><td>Genie 3, Marble</td></tr> <tr><td>A planning engine inside a robot or game agent that predicts what each possible action will do</td><td>V-JEPA 2, Dreamer</td></tr> <tr><td>The rough picture of the world that an LLM may build inside itself while learning to predict text</td><td>The board-game and map studies <a href="https://rajnandan.com/posts/large-language-models-vs-world-models/#does-an-llm-have-a-world-model-inside-it">further down</a></td></tr> </tbody></table> <h3 id="drawing-the-future-or-summarizing-it">Drawing the future or summarizing it<a class="zola-anchor" href="#drawing-the-future-or-summarizing-it" aria-label="Anchor link for: drawing-the-future-or-summarizing-it" style="visibility: hidden;"></a> </h3> <p>World models also differ in what they output.</p> <p>Some draw the future pixel by pixel. Genie 3 generates each video frame from the frames before it and your latest input. NVIDIA's Cosmos models generate video too. People can watch the output, which makes these models useful for producing training footage and test scenes.</p> <p>Others predict a compact summary of what comes next and never draw it. Meta's V-JEPA 2 works this way. It follows an approach Yann LeCun argued for at Meta: much of a scene can't be predicted and doesn't matter for the task at hand. A robot carrying a cup across a room doesn't need to predict how each leaf on the tree outside the window will move.</p> <p>In one test, skipping the pixels paid off in speed. Meta's team compared the two styles on the same robot. Planning with Cosmos, which draws video, took 4 minutes for each arm movement, so moving one object took over an hour. V-JEPA 2 needed 16 seconds per movement and succeeded more often.</p> <h2 id="world-models-you-can-look-at-today">World models you can look at today<a class="zola-anchor" href="#world-models-you-can-look-at-today" aria-label="Anchor link for: world-models-you-can-look-at-today" style="visibility: hidden;"></a> </h2> <table><thead><tr><th>Model</th><th>Made by</th><th>What it does</th><th>Released</th></tr></thead><tbody> <tr><td><a rel="noopener nofollow noreferrer" target="_blank" href="https://deepmind.google/blog/genie-3-a-new-frontier-for-world-models/">Genie 3</a></td><td>Google DeepMind</td><td>Turns a text prompt into a world you can walk through in real time</td><td>Announced August 2025</td></tr> <tr><td><a rel="noopener nofollow noreferrer" target="_blank" href="https://ai.meta.com/blog/v-jepa-2-world-model-benchmarks/">V-JEPA 2</a></td><td>Meta</td><td>Learned from over a million hours of video; helps robot arms plan their moves</td><td>June 2025, free to download</td></tr> <tr><td><a rel="noopener nofollow noreferrer" target="_blank" href="https://nvidianews.nvidia.com/news/nvidia-launches-cosmos-world-foundation-model-platform-to-accelerate-physical-ai-development">Cosmos</a></td><td>NVIDIA</td><td>Makes realistic video for training robots and self-driving cars; learned from 20 million hours of video</td><td>January 2025, free to download</td></tr> <tr><td><a rel="noopener nofollow noreferrer" target="_blank" href="https://www.worldlabs.ai/blog/marble-world-model">Marble</a></td><td>World Labs</td><td>Builds 3D worlds from text, photos, or video that you can save and reuse in games and 3D tools</td><td>November 2025</td></tr> <tr><td><a rel="noopener nofollow noreferrer" target="_blank" href="https://waymo.com/blog/2026/02/the-waymo-world-model-a-new-frontier-for-autonomous-driving-simulation/">Waymo World Model</a></td><td>Waymo, built on Genie 3</td><td>Makes rare driving scenes to test Waymo's cars, as camera images and lidar (laser distance sensor) data</td><td>February 2026</td></tr> <tr><td><a rel="noopener nofollow noreferrer" target="_blank" href="https://arxiv.org/abs/2509.24527">Dreamer 4</a></td><td>Google DeepMind</td><td>Learned Minecraft from recorded play and practiced only inside its own simulation</td><td>September 2025, research paper</td></tr> </tbody></table> <p>Google has also opened a prototype called <a rel="noopener nofollow noreferrer" target="_blank" href="https://blog.google/innovation-and-ai/models-and-research/google-deepmind/project-genie/">Project Genie</a> to Google AI Ultra subscribers in the US. Each generated world runs for up to 60 seconds.</p> <h2 id="the-differences-side-by-side">The differences, side by side<a class="zola-anchor" href="#the-differences-side-by-side" aria-label="Anchor link for: the-differences-side-by-side" style="visibility: hidden;"></a> </h2> <table><thead><tr><th></th><th>Large language model</th><th>World model</th></tr></thead><tbody> <tr><td>Predicts</td><td>The next word</td><td>What happens next in a scene</td></tr> <tr><td>Learns from</td><td>Text and code, plus images and audio in newer models</td><td>Video, sensor data, and game or robot recordings, ideally with the actions taken</td></tr> <tr><td>Role of actions</td><td>None in the main training stage</td><td>Central: "if this happens, what comes next?"</td></tr> <tr><td>Output you see</td><td>Words and code</td><td>Video frames, 3D scenes, or predictions a robot uses</td></tr> <tr><td>Strong at</td><td>Language, knowledge, writing, coding</td><td>Space, motion, cause and effect, planning physical moves</td></tr> <tr><td>Typical failure</td><td>Makes up facts with confidence</td><td>Physics glitches; worlds that drift over time</td></tr> <tr><td>How widely used</td><td>ChatGPT alone reached about 10% of the world's adults by mid-2025</td><td>Mostly research and early products</td></tr> </tbody></table> <p>Three differences in that table do most of the work.</p> <p>The biggest difference is the role of actions. An LLM can tell you what usually happens when someone knocks a glass off a table, because people have written about it. A world model is built to predict what happens when this particular glass gets this particular push, including situations nobody ever wrote down.</p> <p>The training data differs. Text is plentiful, and it already describes the world in words. Video is plentiful too, but most of it doesn't record the actions behind it. A dashcam clip shows the road but not how the driver turned the wheel. Video with the actions recorded is the scarce ingredient for world models, so builders combine a lot of plain video with a little of the rare kind. V-JEPA 2 learned mostly from ordinary video and needed under 62 hours of robot footage with the actions recorded.</p> <p>The feedback differs. During training, an LLM is graded on whether it guessed the next word of a document. A world model trained on video is graded on whether its predicted frame matches the frame the camera recorded.</p> <h2 id="same-job-two-approaches">Same job, two approaches<a class="zola-anchor" href="#same-job-two-approaches" aria-label="Anchor link for: same-job-two-approaches" style="visibility: hidden;"></a> </h2> <h3 id="making-a-game">Making a game<a class="zola-anchor" href="#making-a-game" aria-label="Anchor link for: making-a-game" style="visibility: hidden;"></a> </h3> <p>Ask an LLM for a small browser game and it writes code. The result is precise. The jump height is a number, the walls stay where they are, and the game plays the same way tomorrow.</p> <p>Ask Genie 3 for a game world and it draws the world frame by frame while you move through it. You can type "a snowy mountain village at dusk" and walk around in it, with no code at all. The world does not last long, though. Genie 3 keeps a world consistent for a few minutes and remembers changes you make for up to a minute.</p> <h3 id="moving-a-cup-with-a-robot-arm">Moving a cup with a robot arm<a class="zola-anchor" href="#moving-a-cup-with-a-robot-arm" aria-label="Anchor link for: moving-a-cup-with-a-robot-arm" style="visibility: hidden;"></a> </h3> <p>Give V-JEPA 2 a photo of how the table should look at the end, say with the cup inside the bowl. The robot then plans in a loop:</p> <pre class="z-code"><code><span class="z-text z-plain">goal photo: cup inside the bowl </span><span class="z-text z-plain"> </span><span class="z-text z-plain">1. list a few possible arm movements </span><span class="z-text z-plain">2. predict the result of each one with the world model </span><span class="z-text z-plain">3. pick the movement whose predicted result is closest to the goal </span><span class="z-text z-plain">4. make that one movement for real, look again, repeat </span></code></pre> <p>Meta ran this on robot arms in two labs, without collecting any data from those robots and without training for the specific task. Averaged over the two labs, the arms picked up a cup and put it in the right place 80 percent of the time, and did the same with a box 65 percent of the time.</p> <p>An LLM can write a sensible list of steps for the same job: open the gripper, move above the cup, lower, close, lift. It has no reliable way to tell whether moving three centimeters to the left will knock the cup over. That question is about this cup on this table, and answering it takes a model of the physical scene.</p> <h3 id="mining-diamonds-in-minecraft">Mining diamonds in Minecraft<a class="zola-anchor" href="#mining-diamonds-in-minecraft" aria-label="Anchor link for: mining-diamonds-in-minecraft" style="visibility: hidden;"></a> </h3> <p>Getting a diamond in Minecraft takes a long chain of steps: chop wood, craft tools, mine stone, find iron, smelt it, craft an iron pickaxe, and dig deep enough to find diamonds. The Dreamer 4 team at Google DeepMind set a harder version of the challenge: learn only from about 2,500 hours of recorded human gameplay, with no chance to play the game during training.</p> <p>Dreamer 4 learned a world model of Minecraft from those recordings, then practiced inside it. It became the first agent to get diamonds from recorded data alone. It reached a diamond in 0.7 percent of its hour-long test runs. That rate is low, and the task takes more than 20,000 mouse and keyboard actions to finish.</p> <p>The team also built a second agent on top of Gemma 3, a Google model that understands images and text, the same kind of model that powers chatbots. Dreamer 4 crafted iron pickaxes nearly three times as often as that agent.</p> <h2 id="does-an-llm-have-a-world-model-inside-it">Does an LLM have a world model inside it?<a class="zola-anchor" href="#does-an-llm-have-a-world-model-inside-it" aria-label="Anchor link for: does-an-llm-have-a-world-model-inside-it" style="visibility: hidden;"></a> </h2> <p>This question matters for the whole comparison. If LLMs already carry an accurate picture of the world, separate world models add less. To find out, researchers train language-style models on worlds they understand completely, like a board game, a street map, or a solar system. Then they check whether the model learned the world itself or only patterns that happen to fit the training data. The results point both ways.</p> <h3 id="evidence-for-a-board-it-never-saw">Evidence for: a board it never saw<a class="zola-anchor" href="#evidence-for-a-board-it-never-saw" aria-label="Anchor link for: evidence-for-a-board-it-never-saw" style="visibility: hidden;"></a> </h3> <p>Othello is a board game played on an 8-by-8 grid. In a <a rel="noopener nofollow noreferrer" target="_blank" href="https://arxiv.org/abs/2210.13382">2023 study</a>, researchers trained a small language model on nothing but lists of Othello moves. It never saw a board or the rules, yet it learned to suggest legal moves.</p> <p>When the researchers looked inside the model, they found a picture of the board: they could read off which squares held which pieces. When they changed that inner board, as if moving a piece by hand, the model's suggested moves changed to match. Keeping track of the board helped the model predict moves, so it learned to keep track of the board.</p> <h3 id="evidence-against-a-map-that-doesn-t-exist">Evidence against: a map that doesn't exist<a class="zola-anchor" href="#evidence-against-a-map-that-doesn-t-exist" aria-label="Anchor link for: evidence-against-a-map-that-doesn-t-exist" style="visibility: hidden;"></a> </h3> <p>In a <a rel="noopener nofollow noreferrer" target="_blank" href="https://arxiv.org/abs/2406.03689">2024 study</a>, researchers from Harvard, MIT, and Cornell trained language-style models on New York City taxi trips, written out as turn-by-turn directions. The models got very good at giving directions: 96 to 99 percent of their routes were valid.</p> <p>Then the researchers drew the street map that the models' answers implied. It was full of streets that don't exist, curving across Manhattan and linking intersections far apart.</p> <p>The fake streets mattered as soon as the city changed. Take the model trained on the shortest routes. It produced valid routes 99 percent of the time. When the researchers forced a detour at 1 turn in 100, that dropped to 69 percent. At 1 turn in 10, it dropped to 8 percent.</p> <p>A model trained on aimless random drives, the kind no real taxi takes, held up far better. It still found valid routes 97 percent of the time with half the turns detoured, because it had seen more of the city. The researchers' stricter checks still found flaws in its map.</p> <h3 id="evidence-against-orbits-without-gravity">Evidence against: orbits without gravity<a class="zola-anchor" href="#evidence-against-orbits-without-gravity" aria-label="Anchor link for: evidence-against-orbits-without-gravity" style="visibility: hidden;"></a> </h3> <p>In a <a rel="noopener nofollow noreferrer" target="_blank" href="https://arxiv.org/abs/2507.06952">2025 study</a>, a team from Harvard and MIT trained a model on the movements of planets in 10 million made-up solar systems. It learned to predict where each planet would go next with high accuracy.</p> <p>Then the researchers asked it about the force behind those orbits: the pull of gravity on each planet. They worked out the rule the model was following, and it made no physical sense. The rule also changed from one group of solar systems to the next.</p> <p>In the early 1600s, Johannes Kepler described the shapes of planetary orbits. In 1687, Isaac Newton explained those shapes with a law of gravity. The model had learned something like Kepler's shapes without Newton's law.</p> <p>The team ran a smaller version of the test on three well-known LLMs: OpenAI's o3, Anthropic's Claude Sonnet 4, and Google's Gemini 2.5 Pro. Newton's law says the pull depends on both masses and on the distance between them. The rules behind all three models' answers left the distance out, and each used only one of the two masses. Any of these models will recite Newton's law if you ask.</p> <h3 id="a-twist-from-2026">A twist from 2026<a class="zola-anchor" href="#a-twist-from-2026" aria-label="Anchor link for: a-twist-from-2026" style="visibility: hidden;"></a> </h3> <p>A <a rel="noopener nofollow noreferrer" target="_blank" href="https://arxiv.org/abs/2602.06923">follow-up in 2026</a> by another team showed a way around the problem. When the researchers let the model look only at a planet's last two positions instead of its whole history, the model worked out the pull of gravity. With the whole history in view, it went back to tracing ellipses the way Kepler did.</p> <h3 id="the-short-answer-partly">The short answer: partly<a class="zola-anchor" href="#the-short-answer-partly" aria-label="Anchor link for: the-short-answer-partly" style="visibility: hidden;"></a> </h3> <p>LLMs build pieces of a world model as a side effect of predicting text, and some of those pieces are good. Training only checks whether the pieces help predict the next word. They can be right in common situations and wrong in unusual ones, with no warning sign. World-model research puts the model of the world first and tests it against what happens.</p> <h2 id="use-cases">Use cases<a class="zola-anchor" href="#use-cases" aria-label="Anchor link for: use-cases" style="visibility: hidden;"></a> </h2> <h3 id="llms">LLMs<a class="zola-anchor" href="#llms" aria-label="Anchor link for: llms" style="visibility: hidden;"></a> </h3> <p>A September 2025 <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.nber.org/papers/w34255">study of ChatGPT use</a>, by OpenAI researchers and Harvard economist David Deming, found that three kinds of conversation made up nearly 80 percent of it: practical advice, looking things up, and writing. More than 70 percent of messages had nothing to do with work.</p> <p>Common uses, with an example of each:</p> <table><thead><tr><th>Use</th><th>Example</th></tr></thead><tbody> <tr><td>Writing and editing</td><td>Turning rough notes into a clear email</td></tr> <tr><td>Answering questions</td><td>Explaining a term from a medical report in plain words</td></tr> <tr><td>Coding</td><td>Writing a function, explaining an error, reviewing a pull request</td></tr> <tr><td>Working with documents</td><td>Summarizing a long contract, pulling totals out of invoices</td></tr> <tr><td>Translation</td><td>Turning a help article into Hindi or Spanish</td></tr> <tr><td>Agents</td><td>Fixing a bug across several files by running tools, one step at a time</td></tr> </tbody></table> <p>Coding is a small slice of ChatGPT traffic, but it is the whole job of dedicated tools like GitHub Copilot, Cursor, and Claude Code.</p> <h3 id="world-models">World models<a class="zola-anchor" href="#world-models" aria-label="Anchor link for: world-models" style="visibility: hidden;"></a> </h3> <p>Robots need practice, and practice in the real world is slow and hard on the hardware. World models give robots somewhere else to practice.</p> <ul> <li>NVIDIA offers Cosmos as a way to make training video for robots. At the Cosmos launch, Agility, which builds humanoid robots, said the model lets it create training scenes and rely less on expensive real-world recording.</li> <li>V-JEPA 2 plans robot-arm moves, as in the cup example above.</li> <li>Marble builds rooms for robot simulators. In a <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.worldlabs.ai/case-studies/1-robotics">World Labs case study</a>, a contributor to the RoboSuite simulator said one high-quality simulation environment can take weeks to put together by hand. With Marble, several usable environments took hours.</li> </ul> <p>Self-driving cars face a different data problem. The dangerous moments are rare, so a fleet can drive for a very long time without meeting a tornado or an elephant in the road. Waymo's world model generates those scenes on demand, and engineers can change a scene with a plain-language prompt. In February 2026, Waymo said its cars had driven nearly 200 million fully autonomous miles on real roads and billions of miles in simulation. NVIDIA also named Uber and Waabi among the first companies to adopt Cosmos.</p> <p>Training AI agents is the use DeepMind emphasizes for Genie 3. The team gave its SIMA agent goals inside worlds that Genie 3 generated. SIMA chose the moves, and Genie 3 drew what happened, simulating the results of each move without knowing the goal.</p> <p>Games, film, and design are the most visible uses. Project Genie lets people explore worlds they describe. World Labs shot its Marble launch video on an LED stage with Marble worlds as the backdrops, and a game developer built a rail shooter in Unreal Engine 5 from Marble scenes.</p> <h2 id="they-work-best-together">They work best together<a class="zola-anchor" href="#they-work-best-together" aria-label="Anchor link for: they-work-best-together" style="visibility: hidden;"></a> </h2> <p>The two kinds of model cover each other's gaps. An LLM is good at understanding what you want and breaking it into steps. A world model is good at predicting what a physical step will do. Several real systems already pair them:</p> <ul> <li><a rel="noopener nofollow noreferrer" target="_blank" href="https://deepmind.google/blog/sima-2-an-agent-that-plays-reasons-and-learns-with-you-in-virtual-3d-worlds/">SIMA 2</a>, the next version of that agent, uses Gemini to understand instructions and reason about goals. DeepMind reports that it kept improving through its own practice in Genie 3 worlds it had never seen.</li> <li>Project Genie uses Gemini and Google's Nano Banana Pro image model to help you describe and preview a world before Genie 3 generates it.</li> <li>Meta connected V-JEPA 2 to a language model so the pair can answer questions about videos.</li> </ul> <p>Picture a home robot asked to put a mug in the dishwasher. A language model turns the request into steps: find the mug, open the dishwasher, place the mug on the top rack. For each step, a world model tries a few possible arm movements in simulation and picks the one that ends with the mug on the rack instead of the floor.</p> <h2 id="the-limits-of-world-models-today">The limits of world models today<a class="zola-anchor" href="#the-limits-of-world-models-today" aria-label="Anchor link for: the-limits-of-world-models-today" style="visibility: hidden;"></a> </h2> <p>World models are early, and their builders are open about the limits. DeepMind's own list for Genie 3:</p> <ul> <li>Worlds hold together for a few minutes of continuous interaction.</li> <li>Agents can take only a limited range of actions inside it.</li> <li>Several characters acting on their own in one shared world is still an unsolved research problem.</li> <li>It can't recreate real places with perfect accuracy.</li> <li>It tends to render readable text only when that text appears in the prompt.</li> </ul> <p>Two broader problems apply across the field. Agents trained inside a world model exploit its mistakes, as the 2018 Doom agent did with the fireballs. And a model can look right while being wrong. A generated video can look real and still break physics, and the taxi and orbit studies show that accurate predictions can hide a wrong model underneath. The test that counts is whether plans made inside the model work in the real world.</p> <h2 id="a-quick-test-for-ai-headlines">A quick test for AI headlines<a class="zola-anchor" href="#a-quick-test-for-ai-headlines" aria-label="Anchor link for: a-quick-test-for-ai-headlines" style="visibility: hidden;"></a> </h2> <p>Three questions sort most announcements.</p> <ol> <li>What does the model predict? The next word points to an LLM. The next frame, sensor reading, or state points to a world model.</li> <li>Can something act inside it and see the consequences? If a person or an agent can take actions and the model responds to each one, it behaves like a world model. A video generator that plays one clip from one prompt does not meet that bar.</li> <li>How was it checked? Look for tests against real outcomes, such as a robot that finished the task or a plan that worked outside the simulation. A realistic-looking video, on its own, says little about the physics inside the model.</li> </ol> <h2 id="further-reading">Further reading<a class="zola-anchor" href="#further-reading" aria-label="Anchor link for: further-reading" style="visibility: hidden;"></a> </h2> <ul> <li>Google DeepMind, <a rel="noopener nofollow noreferrer" target="_blank" href="https://deepmind.google/blog/genie-3-a-new-frontier-for-world-models/">Genie 3: A new frontier for world models</a></li> <li>Meta AI, <a rel="noopener nofollow noreferrer" target="_blank" href="https://ai.meta.com/blog/v-jepa-2-world-model-benchmarks/">Introducing the V-JEPA 2 world model</a>, and the <a rel="noopener nofollow noreferrer" target="_blank" href="https://arxiv.org/abs/2506.09985">V-JEPA 2 paper</a></li> <li>David Ha and Jürgen Schmidhuber, <a rel="noopener nofollow noreferrer" target="_blank" href="https://worldmodels.github.io/">World Models</a> (2018)</li> <li>Danijar Hafner and colleagues, <a rel="noopener nofollow noreferrer" target="_blank" href="https://arxiv.org/abs/2509.24527">Training Agents Inside of Scalable World Models</a> (Dreamer 4)</li> <li>Keyon Vafa and colleagues, <a rel="noopener nofollow noreferrer" target="_blank" href="https://arxiv.org/abs/2406.03689">Evaluating the World Model Implicit in a Generative Model</a> (the taxi map)</li> <li>Keyon Vafa and colleagues, <a rel="noopener nofollow noreferrer" target="_blank" href="https://arxiv.org/abs/2507.06952">What Has a Foundation Model Found?</a> (the orbits)</li> <li>Ziming Liu and colleagues, <a rel="noopener nofollow noreferrer" target="_blank" href="https://arxiv.org/abs/2602.06923">From Kepler to Newton</a></li> <li>Kenneth Li and colleagues, <a rel="noopener nofollow noreferrer" target="_blank" href="https://arxiv.org/abs/2210.13382">Emergent World Representations</a> (Othello-GPT), with a readable summary on <a rel="noopener nofollow noreferrer" target="_blank" href="https://thegradient.pub/othello/">The Gradient</a></li> <li>Waymo, <a rel="noopener nofollow noreferrer" target="_blank" href="https://waymo.com/blog/2026/02/the-waymo-world-model-a-new-frontier-for-autonomous-driving-simulation/">The Waymo World Model</a></li> <li>NVIDIA, <a rel="noopener nofollow noreferrer" target="_blank" href="https://nvidianews.nvidia.com/news/nvidia-launches-cosmos-world-foundation-model-platform-to-accelerate-physical-ai-development">Cosmos launch announcement</a></li> <li>World Labs, <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.worldlabs.ai/blog/marble-world-model">Marble: A Multimodal World Model</a></li> <li>Aaron Chatterji and colleagues, <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.nber.org/papers/w34255">How People Use ChatGPT</a> (NBER, 2025)</li> <li>Ashish Vaswani and colleagues, <a rel="noopener nofollow noreferrer" target="_blank" href="https://arxiv.org/abs/1706.03762">Attention Is All You Need</a> (2017)</li> </ul> 2026-08-30T00:00:00+00:00 2026-08-30T00:00:00+00:00 https://rajnandan.com/posts/introducing-json-to-md/ <p>I built <a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/rajnandan1/json-to-md">json-to-md</a> to turn a JSON document into Markdown that a person can read and a model can follow. It runs in Node, in the browser, in Go, and on the command line. The TypeScript and Go implementations produce byte-identical output, and a shared corpus plus a cross-implementation fuzz gate in CI keep them that way.</p> <p>An order object going through it:</p> <pre data-lang="json" class="language-json z-code"><code class="language-json" data-lang="json"><span class="z-source z-json"><span class="z-meta z-mapping z-json"><span class="z-punctuation z-section z-mapping z-begin z-json">{</span> </span></span><span class="z-source z-json"><span class="z-meta z-mapping z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>order<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-json"><span class="z-punctuation z-section z-mapping z-begin z-json">{</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>id<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>ord_1042<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>status<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>paid<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>customer<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-json"><span class="z-punctuation z-section z-mapping z-begin z-json">{</span> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>name<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>Asha<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>email<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>asha@example.com<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span> <span class="z-punctuation z-section z-mapping z-end z-json">}</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>items<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-punctuation z-section z-sequence z-begin z-json">[</span> </span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"> <span class="z-meta z-mapping z-json"><span class="z-punctuation z-section z-mapping z-begin z-json">{</span> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>sku<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>A1<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>qty<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-constant z-numeric z-integer z-decimal z-json">2</span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>price<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-constant z-numeric z-float z-decimal z-json">499<span class="z-punctuation z-separator z-decimal z-json">.</span>00</span> <span class="z-punctuation z-section z-mapping z-end z-json">}</span></span><span class="z-punctuation z-separator z-sequence z-json">,</span> </span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"> <span class="z-meta z-mapping z-json"><span class="z-punctuation z-section z-mapping z-begin z-json">{</span> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>sku<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>B7<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>qty<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-constant z-numeric z-integer z-decimal z-json">1</span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>price<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-constant z-numeric z-float z-decimal z-json">1299<span class="z-punctuation z-separator z-decimal z-json">.</span>50</span> <span class="z-punctuation z-section z-mapping z-end z-json">}</span></span> </span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"> <span class="z-punctuation z-section z-sequence z-end z-json">]</span></span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> <span class="z-punctuation z-section z-mapping z-end z-json">}</span></span> </span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-section z-mapping z-end z-json">}</span></span> </span></code></pre> <pre data-lang="markdown" class="language-markdown z-code"><code class="language-markdown" data-lang="markdown"><span class="z-text z-html z-markdown"><span class="z-meta z-block-level z-markdown"><span class="z-markup z-heading z-1 z-markdown"><span class="z-punctuation z-definition z-heading z-begin z-markdown">#</span> </span><span class="z-markup z-heading z-1 z-markdown"><span class="z-entity z-name z-section z-markdown">Results</span><span class="z-meta z-whitespace z-newline z-markdown"> </span></span></span></span><span class="z-text z-html z-markdown"> </span><span class="z-text z-html z-markdown"><span class="z-meta z-block-level z-markdown"><span class="z-markup z-heading z-2 z-markdown"><span class="z-punctuation z-definition z-heading z-begin z-markdown">##</span> </span><span class="z-markup z-heading z-2 z-markdown"><span class="z-entity z-name z-section z-markdown">order</span><span class="z-meta z-whitespace z-newline z-markdown"> </span></span></span></span><span class="z-text z-html z-markdown"> </span><span class="z-text z-html z-markdown"><span class="z-meta z-block-level z-markdown"><span class="z-markup z-heading z-markdown"><span class="z-punctuation z-definition z-heading z-begin z-markdown">###</span> </span><span class="z-markup z-heading z-markdown"><span class="z-entity z-name z-section z-markdown">id</span><span class="z-meta z-whitespace z-newline z-markdown"> </span></span></span></span><span class="z-text z-html z-markdown"> </span><span class="z-text z-html z-markdown"><span class="z-meta z-paragraph z-markdown">ord<span class="z-constant z-character z-escape z-markdown">\_</span>1042 </span></span><span class="z-text z-html z-markdown"><span class="z-meta z-paragraph z-markdown"> </span></span><span class="z-text z-html z-markdown"><span class="z-meta z-block-level z-markdown"><span class="z-markup z-heading z-markdown"><span class="z-punctuation z-definition z-heading z-begin z-markdown">###</span> </span><span class="z-markup z-heading z-markdown"><span class="z-entity z-name z-section z-markdown">status</span><span class="z-meta z-whitespace z-newline z-markdown"> </span></span></span></span><span class="z-text z-html z-markdown"> </span><span class="z-text z-html z-markdown"><span class="z-meta z-paragraph z-markdown">paid </span></span><span class="z-text z-html z-markdown"><span class="z-meta z-paragraph z-markdown"> </span></span><span class="z-text z-html z-markdown"><span class="z-meta z-block-level z-markdown"><span class="z-markup z-heading z-markdown"><span class="z-punctuation z-definition z-heading z-begin z-markdown">###</span> </span><span class="z-markup z-heading z-markdown"><span class="z-entity z-name z-section z-markdown">customer</span><span class="z-meta z-whitespace z-newline z-markdown"> </span></span></span></span><span class="z-text z-html z-markdown"> </span><span class="z-text z-html z-markdown"><span class="z-meta z-block-level z-markdown"><span class="z-markup z-heading z-markdown"><span class="z-punctuation z-definition z-heading z-begin z-markdown">####</span> </span><span class="z-markup z-heading z-markdown"><span class="z-entity z-name z-section z-markdown">name</span><span class="z-meta z-whitespace z-newline z-markdown"> </span></span></span></span><span class="z-text z-html z-markdown"> </span><span class="z-text z-html z-markdown"><span class="z-meta z-paragraph z-markdown">Asha </span></span><span class="z-text z-html z-markdown"><span class="z-meta z-paragraph z-markdown"> </span></span><span class="z-text z-html z-markdown"><span class="z-meta z-block-level z-markdown"><span class="z-markup z-heading z-markdown"><span class="z-punctuation z-definition z-heading z-begin z-markdown">####</span> </span><span class="z-markup z-heading z-markdown"><span class="z-entity z-name z-section z-markdown">email</span><span class="z-meta z-whitespace z-newline z-markdown"> </span></span></span></span><span class="z-text z-html z-markdown"> </span><span class="z-text z-html z-markdown"><span class="z-meta z-paragraph z-markdown"><span class="z-markup z-underline z-link z-markdown">asha@example.com</span> </span></span><span class="z-text z-html z-markdown"><span class="z-meta z-paragraph z-markdown"> </span></span><span class="z-text z-html z-markdown"><span class="z-meta z-block-level z-markdown"><span class="z-markup z-heading z-markdown"><span class="z-punctuation z-definition z-heading z-begin z-markdown">###</span> </span><span class="z-markup z-heading z-markdown"><span class="z-entity z-name z-section z-markdown">items</span><span class="z-meta z-whitespace z-newline z-markdown"> </span></span></span></span><span class="z-text z-html z-markdown"> </span><span class="z-text z-html z-markdown"><span class="z-meta z-block-level z-markdown"><span class="z-meta z-table z-header z-markdown-gfm"><span class="z-punctuation z-separator z-table-cell z-markdown">|</span> sku <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> qty <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> price <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> </span></span></span><span class="z-text z-html z-markdown"><span class="z-meta z-block-level z-markdown"><span class="z-meta z-table z-header-separator z-markdown-gfm"><span class="z-punctuation z-separator z-table-cell z-markdown">|</span> <span class="z-punctuation z-section z-table-header z-markdown">---</span> <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> <span class="z-punctuation z-section z-table-header z-markdown">---</span> <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> <span class="z-punctuation z-section z-table-header z-markdown">---</span> <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> </span><span class="z-meta z-table z-markdown-gfm"></span></span></span><span class="z-text z-html z-markdown"><span class="z-meta z-block-level z-markdown"><span class="z-meta z-table z-markdown-gfm"><span class="z-punctuation z-separator z-table-cell z-markdown">|</span> A1 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> 2 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> 499.00 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> </span></span></span><span class="z-text z-html z-markdown"><span class="z-meta z-block-level z-markdown"><span class="z-meta z-table z-markdown-gfm"><span class="z-punctuation z-separator z-table-cell z-markdown">|</span> B7 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> 1 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> 1299.50 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> </span></span></span></code></pre> <p>Object keys become headings. Arrays of objects become tables. Strings come out escaped, so <code>ord_1042</code> renders as <code>ord\_1042</code>, and a value containing <code>&lt;script&gt;</code> or <code>**bold**</code> shows up as literal text. Data never injects formatting into the document. The prices kept their spelling too: <code>499.00</code> stayed <code>499.00</code> because the text entry point preserves each number's original lexeme instead of round-tripping it through a float.</p> <h2 id="why-i-needed-this">Why I needed this<a class="zola-anchor" href="#why-i-needed-this" aria-label="Anchor link for: why-i-needed-this" style="visibility: hidden;"></a> </h2> <p>JSON keeps landing in places where a person or a model reads it:</p> <ol> <li>Prompts. Tool results and retrieved records go into the context window, and the model reads <code>{</code>, <code>"</code>, and <code>,</code> before it reads any data.</li> <li>Agent logs and memory files. You will not open a 400-line escaped JSON blob to check what an agent did.</li> <li>Docs and PR descriptions. Pasting an API response means someone reformats it by hand.</li> </ol> <p>Markdown fixes the reading problem. Before I would trust a converter inside a pipeline, I wanted four guarantees from it, so I wrote them down first and built to them.</p> <ul> <li><strong>Deterministic.</strong> Same input, same bytes. LF line endings, one blank line between blocks, no trailing spaces, one final newline.</li> <li><strong>One contract, two languages.</strong> The TypeScript and Go code share a corpus of expected outputs, and CI fuzzes both implementations against each other on every change.</li> <li><strong>Safe by construction.</strong> Every string is escaped. A value cannot open a heading, a link, or an HTML tag.</li> <li><strong>One-way.</strong> The output is a projection for reading. When a downstream step must parse, validate, or store the data, keep the JSON.</li> </ul> <h2 id="how-values-render">How values render<a class="zola-anchor" href="#how-values-render" aria-label="Anchor link for: how-values-render" style="visibility: hidden;"></a> </h2> <table><thead><tr><th>Input</th><th>Rendering</th></tr></thead><tbody> <tr><td>Object keys</td><td>Headings H2 through H6, then nested lists once nesting passes H6</td></tr> <tr><td>Array of objects</td><td>One GFM table</td></tr> <tr><td>Any other array</td><td>Unordered list, in source order</td></tr> <tr><td>Non-empty object or array inside a table cell</td><td>Link to a detail section headed by the value's JSON Pointer</td></tr> <tr><td>String</td><td>Escaped literal text</td></tr> <tr><td>Whole-string absolute <code>http(s)</code> URL</td><td>Markdown link</td></tr> <tr><td><code>null</code>, <code>""</code>, <code>[]</code>, <code>{}</code></td><td>Inline code, each kept distinct from a missing cell</td></tr> </tbody></table> <p>Tables are where the format pays for itself. Three rows of the classic <code>cars.json</code> dataset:</p> <pre data-lang="markdown" class="language-markdown z-code"><code class="language-markdown" data-lang="markdown"><span class="z-text z-html z-markdown"><span class="z-meta z-block-level z-markdown"><span class="z-markup z-heading z-1 z-markdown"><span class="z-punctuation z-definition z-heading z-begin z-markdown">#</span> </span><span class="z-markup z-heading z-1 z-markdown"><span class="z-entity z-name z-section z-markdown">Cars</span><span class="z-meta z-whitespace z-newline z-markdown"> </span></span></span></span><span class="z-text z-html z-markdown"> </span><span class="z-text z-html z-markdown"><span class="z-meta z-block-level z-markdown"><span class="z-meta z-table z-header z-markdown-gfm"><span class="z-punctuation z-separator z-table-cell z-markdown">|</span> Name <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> Miles<span class="z-constant z-character z-escape z-markdown">\_</span>per<span class="z-constant z-character z-escape z-markdown">\_</span>Gallon <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> Cylinders <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> Displacement <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> Horsepower <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> Weight<span class="z-constant z-character z-escape z-markdown">\_</span>in<span class="z-constant z-character z-escape z-markdown">\_</span>lbs <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> Acceleration <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> Year <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> Origin <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> </span></span></span><span class="z-text z-html z-markdown"><span class="z-meta z-block-level z-markdown"><span class="z-meta z-table z-header-separator z-markdown-gfm"><span class="z-punctuation z-separator z-table-cell z-markdown">|</span> <span class="z-punctuation z-section z-table-header z-markdown">---</span> <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> <span class="z-punctuation z-section z-table-header z-markdown">---</span> <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> <span class="z-punctuation z-section z-table-header z-markdown">---</span> <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> <span class="z-punctuation z-section z-table-header z-markdown">---</span> <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> <span class="z-punctuation z-section z-table-header z-markdown">---</span> <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> <span class="z-punctuation z-section z-table-header z-markdown">---</span> <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> <span class="z-punctuation z-section z-table-header z-markdown">---</span> <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> <span class="z-punctuation z-section z-table-header z-markdown">---</span> <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> <span class="z-punctuation z-section z-table-header z-markdown">---</span> <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> </span><span class="z-meta z-table z-markdown-gfm"></span></span></span><span class="z-text z-html z-markdown"><span class="z-meta z-block-level z-markdown"><span class="z-meta z-table z-markdown-gfm"><span class="z-punctuation z-separator z-table-cell z-markdown">|</span> chevrolet chevelle malibu <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> 18 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> 8 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> 307 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> 130 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> 3504 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> 12 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> 1970-01-01 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> USA <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> </span></span></span><span class="z-text z-html z-markdown"><span class="z-meta z-block-level z-markdown"><span class="z-meta z-table z-markdown-gfm"><span class="z-punctuation z-separator z-table-cell z-markdown">|</span> buick skylark 320 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> 15 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> 8 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> 350 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> 165 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> 3693 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> 11.5 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> 1970-01-01 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> USA <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> </span></span></span><span class="z-text z-html z-markdown"><span class="z-meta z-block-level z-markdown"><span class="z-meta z-table z-markdown-gfm"><span class="z-punctuation z-separator z-table-cell z-markdown">|</span> plymouth satellite <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> 18 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> 8 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> 318 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> 150 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> 3436 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> 11 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> 1970-01-01 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> USA <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> </span></span></span></code></pre> <p>Pretty-printed, those three records cost 265 tokens on the <code>o200k_base</code> tokenizer. Minified, 173. The table, 168. Each key name appears once in the header row instead of once per record, and the gap widens with every row you add.</p> <p>There is no input-size, nesting-depth, or table-size limit in the converter. Parsing, validation, and rendering use explicit stacks instead of recursion, so a deeply nested document cannot overflow the call stack.</p> <h2 id="install">Install<a class="zola-anchor" href="#install" aria-label="Anchor link for: install" style="visibility: hidden;"></a> </h2> <h3 id="node">Node<a class="zola-anchor" href="#node" aria-label="Anchor link for: node" style="visibility: hidden;"></a> </h3> <pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">npm</span></span><span class="z-meta z-function-call z-arguments z-shell"> install @rajnandan1/json-to-md</span> </span></code></pre> <p>Ships ESM and CommonJS builds with TypeScript declarations and no runtime dependencies. Node 18 or newer.</p> <h3 id="browser">Browser<a class="zola-anchor" href="#browser" aria-label="Anchor link for: browser" style="visibility: hidden;"></a> </h3> <p>The same package. Bundle it, or load the release build from a CDN:</p> <pre data-lang="html" class="language-html z-code"><code class="language-html" data-lang="html"><span class="z-text z-html z-basic"><span class="z-meta z-tag z-script z-begin z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;</span><span class="z-entity z-name z-tag z-script z-html">script</span></span><span class="z-meta z-tag z-script z-begin z-html"> <span class="z-meta z-attribute-with-value z-html"><span class="z-entity z-other z-attribute-name z-html">type</span></span></span><span class="z-meta z-tag z-script z-begin z-html"><span class="z-meta z-attribute-with-value z-html"><span class="z-punctuation z-separator z-key-value z-html">=</span></span></span><span class="z-meta z-tag z-script z-begin z-html"><span class="z-meta z-attribute-with-value z-html"></span></span><span class="z-meta z-tag z-script z-begin z-html"><span class="z-meta z-attribute-with-value z-html"><span class="z-string z-quoted z-double z-html"><span class="z-punctuation z-definition z-string z-begin z-html">&quot;</span>module<span class="z-punctuation z-definition z-string z-end z-html">&quot;</span></span></span><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span> </span><span class="z-text z-html z-basic"> <span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-import z-js"><span class="z-keyword z-control z-import-export z-js">import</span> </span><span class="z-meta z-import z-js"><span class="z-meta z-block z-js"><span class="z-punctuation z-section z-block z-begin z-js">{</span> <span class="z-variable z-other z-readwrite z-js">convertJsonText</span> <span class="z-punctuation z-section z-block z-end z-js">}</span></span> <span class="z-keyword z-control z-import-export z-js">from</span> <span class="z-meta z-string z-js"><span class="z-string z-quoted z-double z-js"><span class="z-punctuation z-definition z-string z-begin z-js">&quot;</span>https://cdn.jsdelivr.net/npm/@rajnandan1/json-to-md@3/dist/index.js<span class="z-punctuation z-definition z-string z-end z-js">&quot;</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span></span> </span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"> <span class="z-support z-type z-object z-dom z-js">document</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-meta z-property z-object z-js">body</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-meta z-property z-object z-js">textContent</span> <span class="z-keyword z-operator z-assignment z-js">=</span> <span class="z-meta z-function-call z-js"><span class="z-variable z-function z-js">convertJsonText</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-meta z-string z-js"><span class="z-string z-quoted z-single z-js"><span class="z-punctuation z-definition z-string z-begin z-js">&#39;</span>{&quot;hello&quot;:&quot;world&quot;}<span class="z-punctuation z-definition z-string z-end z-js">&#39;</span></span></span><span class="z-punctuation z-section z-group z-end z-js">)</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> </span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"></span></span><span class="z-meta z-tag z-script z-end z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;/</span><span class="z-entity z-name z-tag z-script z-html">script</span><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span> </span></code></pre> <p>An IIFE build at <code>dist/index.global.js</code> exposes a <code>jsonToMd</code> global for classic script tags. Pin the version and add a Subresource Integrity hash if you go that route.</p> <h3 id="go">Go<a class="zola-anchor" href="#go" aria-label="Anchor link for: go" style="visibility: hidden;"></a> </h3> <pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">go</span></span><span class="z-meta z-function-call z-arguments z-shell"> get github.com/rajnandan1/json-to-md/go/v3</span> </span></code></pre> <h3 id="cli">CLI<a class="zola-anchor" href="#cli" aria-label="Anchor link for: cli" style="visibility: hidden;"></a> </h3> <pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">brew</span></span><span class="z-meta z-function-call z-arguments z-shell"> tap rajnandan1/homebrew-rajnandan</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">brew</span></span><span class="z-meta z-function-call z-arguments z-shell"> trust rajnandan1/rajnandan <span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> once per machine</span><span class="z-comment z-line z-number-sign z-shell"> </span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">brew</span></span><span class="z-meta z-function-call z-arguments z-shell"> install json-to-md</span> </span></code></pre> <p>Or through the Go toolchain:</p> <pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">go</span></span><span class="z-meta z-function-call z-arguments z-shell"> install github.com/rajnandan1/json-to-md/go/v3/cmd/json-to-md@latest</span> </span></code></pre> <h2 id="use">Use<a class="zola-anchor" href="#use" aria-label="Anchor link for: use" style="visibility: hidden;"></a> </h2> <h3 id="typescript-and-javascript">TypeScript and JavaScript<a class="zola-anchor" href="#typescript-and-javascript" aria-label="Anchor link for: typescript-and-javascript" style="visibility: hidden;"></a> </h3> <pre data-lang="ts" class="language-ts z-code"><code class="language-ts" data-lang="ts"><span class="z-source z-ts"><span class="z-meta z-import z-ts"><span class="z-keyword z-control z-import z-ts">import</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-variable z-other z-readwrite z-alias z-ts">convertJsonText</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-alias z-ts">convertJsonValue</span> <span class="z-punctuation z-definition z-block z-ts">}</span></span> <span class="z-keyword z-control z-from z-ts">from</span> <span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>@rajnandan1/json-to-md<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">convertJsonText</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>{ &quot;hello&quot;: &quot;world&quot; }<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> <span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> untrusted serialized JSON text</span> </span><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">convertJsonValue</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">hello</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>world<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> <span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> already-parsed data you trust</span> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> # Results</span> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> ## hello</span> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> world</span> </span></code></pre> <p>Pick the entry point by where the data came from. <code>convertJsonText</code> takes serialized JSON, parses it without running caller code, rejects duplicate member names, and keeps each number's spelling. <code>convertJsonValue</code> takes a parsed value and validates it at runtime, rejecting cycles, sparse arrays, <code>undefined</code>, <code>BigInt</code>, <code>NaN</code>, functions, <code>Date</code>, <code>Map</code>, <code>Set</code>, and class instances.</p> <p>The difference shows on large integers:</p> <pre data-lang="ts" class="language-ts z-code"><code class="language-ts" data-lang="ts"><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">convertJsonText</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>9007199254740993<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> <span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> 9007199254740993 (preserved)</span> </span><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">convertJsonValue</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-constant z-numeric z-decimal z-ts">9007199254740993</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> <span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> 9007199254740992 (JS rounded it before you called)</span> </span></code></pre> <p>Two options cover the output shape:</p> <pre data-lang="ts" class="language-ts z-code"><code class="language-ts" data-lang="ts"><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">convertJsonValue</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">data</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">heading</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>Orders<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> <span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> custom H1</span> </span><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">convertJsonValue</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">data</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">heading</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-constant z-language z-null z-ts">null</span> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> <span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> no H1</span> </span><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">convertJsonValue</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">data</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">showTypes</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-constant z-language z-boolean z-true z-ts">true</span> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> <span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> annotate values with their JSON type</span> </span></code></pre> <p>With <code>showTypes</code>, scalars get a trailing <code> *(string)*</code>, <code> *(integer)*</code>, <code> *(number)*</code>, or <code> *(boolean)*</code>, and a table column gets the annotation in its header when every cell shares one type. The order document from the top, with types on and the H1 off:</p> <pre data-lang="markdown" class="language-markdown z-code"><code class="language-markdown" data-lang="markdown"><span class="z-text z-html z-markdown"><span class="z-meta z-block-level z-markdown"><span class="z-markup z-heading z-2 z-markdown"><span class="z-punctuation z-definition z-heading z-begin z-markdown">##</span> </span><span class="z-markup z-heading z-2 z-markdown"><span class="z-entity z-name z-section z-markdown">order</span><span class="z-meta z-whitespace z-newline z-markdown"> </span></span></span></span><span class="z-text z-html z-markdown"> </span><span class="z-text z-html z-markdown"><span class="z-meta z-block-level z-markdown"><span class="z-markup z-heading z-markdown"><span class="z-punctuation z-definition z-heading z-begin z-markdown">###</span> </span><span class="z-markup z-heading z-markdown"><span class="z-entity z-name z-section z-markdown">id</span><span class="z-meta z-whitespace z-newline z-markdown"> </span></span></span></span><span class="z-text z-html z-markdown"> </span><span class="z-text z-html z-markdown"><span class="z-meta z-paragraph z-markdown">ord<span class="z-constant z-character z-escape z-markdown">\_</span>1042 <span class="z-markup z-italic z-markdown"><span class="z-punctuation z-definition z-italic z-begin z-markdown">*</span>(string)<span class="z-punctuation z-definition z-italic z-end z-markdown">*</span></span> </span></span><span class="z-text z-html z-markdown"><span class="z-meta z-paragraph z-markdown"> </span></span><span class="z-text z-html z-markdown"><span class="z-meta z-block-level z-markdown"><span class="z-markup z-heading z-markdown"><span class="z-punctuation z-definition z-heading z-begin z-markdown">###</span> </span><span class="z-markup z-heading z-markdown"><span class="z-entity z-name z-section z-markdown">status</span><span class="z-meta z-whitespace z-newline z-markdown"> </span></span></span></span><span class="z-text z-html z-markdown"> </span><span class="z-text z-html z-markdown"><span class="z-meta z-paragraph z-markdown">paid <span class="z-markup z-italic z-markdown"><span class="z-punctuation z-definition z-italic z-begin z-markdown">*</span>(string)<span class="z-punctuation z-definition z-italic z-end z-markdown">*</span></span> </span></span><span class="z-text z-html z-markdown"><span class="z-meta z-paragraph z-markdown"> </span></span><span class="z-text z-html z-markdown"><span class="z-meta z-block-level z-markdown"><span class="z-markup z-heading z-markdown"><span class="z-punctuation z-definition z-heading z-begin z-markdown">###</span> </span><span class="z-markup z-heading z-markdown"><span class="z-entity z-name z-section z-markdown">items</span><span class="z-meta z-whitespace z-newline z-markdown"> </span></span></span></span><span class="z-text z-html z-markdown"> </span><span class="z-text z-html z-markdown"><span class="z-meta z-block-level z-markdown"><span class="z-meta z-table z-header z-markdown-gfm"><span class="z-punctuation z-separator z-table-cell z-markdown">|</span> sku <span class="z-markup z-italic z-markdown"><span class="z-punctuation z-definition z-italic z-begin z-markdown">*</span>(string)<span class="z-punctuation z-definition z-italic z-end z-markdown">*</span></span> <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> qty <span class="z-markup z-italic z-markdown"><span class="z-punctuation z-definition z-italic z-begin z-markdown">*</span>(integer)<span class="z-punctuation z-definition z-italic z-end z-markdown">*</span></span> <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> price <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> </span></span></span><span class="z-text z-html z-markdown"><span class="z-meta z-block-level z-markdown"><span class="z-meta z-table z-header-separator z-markdown-gfm"><span class="z-punctuation z-separator z-table-cell z-markdown">|</span> <span class="z-punctuation z-section z-table-header z-markdown">---</span> <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> <span class="z-punctuation z-section z-table-header z-markdown">---</span> <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> <span class="z-punctuation z-section z-table-header z-markdown">---</span> <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> </span><span class="z-meta z-table z-markdown-gfm"></span></span></span><span class="z-text z-html z-markdown"><span class="z-meta z-block-level z-markdown"><span class="z-meta z-table z-markdown-gfm"><span class="z-punctuation z-separator z-table-cell z-markdown">|</span> A1 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> 2 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> 499.00 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> </span></span></span><span class="z-text z-html z-markdown"><span class="z-meta z-block-level z-markdown"><span class="z-meta z-table z-markdown-gfm"><span class="z-punctuation z-separator z-table-cell z-markdown">|</span> B7 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> 1 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> 1299.50 <span class="z-punctuation z-separator z-table-cell z-markdown">|</span> </span></span></span></code></pre> <p><code>price</code> gets no header annotation because one cell is an integer and the other is not. A literal <code>*</code> in your data is always escaped, so the <code>*(…)*</code> token cannot be forged.</p> <p>Conversion fails at the first error and never returns partial Markdown:</p> <pre data-lang="ts" class="language-ts z-code"><code class="language-ts" data-lang="ts"><span class="z-source z-ts"><span class="z-keyword z-control z-trycatch z-ts">try</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">convertJsonText</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>{&quot;name&quot;:&quot;a&quot;,&quot;name&quot;:&quot;b&quot;}<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span> <span class="z-keyword z-control z-trycatch z-ts">catch</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">e</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">e</span> <span class="z-keyword z-operator z-expression z-instanceof z-ts">instanceof</span> <span class="z-entity z-name z-type z-ts">JsonToMarkdownError</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-other z-object z-ts">e</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-dom z-ts">code</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> <span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> &quot;DUPLICATE_MEMBER_NAME&quot;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-other z-object z-ts">e</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-dom z-ts">location</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> <span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> { offset, line, column }</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-other z-object z-ts">e</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">firstLocation</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> <span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> where the first &quot;name&quot; was</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-other z-object z-ts">e</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">pointer</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> <span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> JSON Pointer, when the bad value is locatable</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></code></pre> <p>Codes: <code>INVALID_JSON_SYNTAX</code>, <code>DUPLICATE_MEMBER_NAME</code>, <code>INVALID_PARSED_VALUE</code>, <code>CYCLIC_REFERENCE</code>, <code>SPARSE_ARRAY</code>, <code>INVALID_OPTION</code>.</p> <h3 id="go-1">Go<a class="zola-anchor" href="#go-1" aria-label="Anchor link for: go-1" style="visibility: hidden;"></a> </h3> <pre data-lang="go" class="language-go z-code"><code class="language-go" data-lang="go"><span class="z-source z-go"><span class="z-keyword z-other z-import z-go">import</span> <span class="z-variable z-other z-go">jsontomd</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>github.com/rajnandan1/json-to-md/go/v3<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span><span class="z-source z-go"> </span><span class="z-source z-go"><span class="z-variable z-declaration z-go">md</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-declaration z-go">err</span> <span class="z-keyword z-operator z-assignment z-go">:=</span> <span class="z-variable z-other z-go">jsontomd</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">ConvertText</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">src</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-comment z-line z-go"><span class="z-punctuation z-definition z-comment z-go">//</span> byte-identical to convertJsonText </span></span><span class="z-source z-go"><span class="z-variable z-other z-go">md</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">err</span> <span class="z-keyword z-operator z-assignment z-go">=</span> <span class="z-variable z-other z-go">jsontomd</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">ConvertValue</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">v</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-comment z-line z-go"><span class="z-punctuation z-definition z-comment z-go">//</span> json.Marshal(v), then the same core </span></span><span class="z-source z-go"><span class="z-variable z-other z-go">md</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">err</span> <span class="z-keyword z-operator z-assignment z-go">=</span> <span class="z-variable z-other z-go">jsontomd</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">ConvertText</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">src</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">jsontomd</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">WithHeading</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>Orders<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span><span class="z-source z-go"><span class="z-variable z-other z-go">md</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">err</span> <span class="z-keyword z-operator z-assignment z-go">=</span> <span class="z-variable z-other z-go">jsontomd</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">ConvertText</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">src</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">jsontomd</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">WithoutHeading</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span><span class="z-source z-go"><span class="z-variable z-other z-go">md</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">err</span> <span class="z-keyword z-operator z-assignment z-go">=</span> <span class="z-variable z-other z-go">jsontomd</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">ConvertText</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">src</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-variable z-other z-go">jsontomd</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">WithTypes</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-punctuation z-section z-parens z-end z-go">)</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span><span class="z-source z-go"> </span><span class="z-source z-go"><span class="z-storage z-type z-keyword z-var z-go">var</span> <span class="z-variable z-declaration z-go">convErr</span> <span class="z-keyword z-operator z-go">*</span><span class="z-variable z-other z-go">jsontomd</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-storage z-type z-go">Error</span> </span><span class="z-source z-go"><span class="z-keyword z-control z-go">if</span> <span class="z-variable z-other z-go">errors</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-function z-go">As</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">err</span><span class="z-punctuation z-separator z-go">,</span> <span class="z-keyword z-operator z-go">&amp;</span><span class="z-variable z-other z-go">convErr</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-comment z-line z-go"><span class="z-punctuation z-definition z-comment z-go">//</span> convErr.Code, .Pointer, .Location: same codes and UTF-16 locations as the TS errors </span></span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></code></pre> <p><code>ConvertValue</code> renders struct fields in declaration order and map keys sorted, because that is what <code>json.Marshal</code> does. The library package has no dependencies.</p> <h3 id="cli-1">CLI<a class="zola-anchor" href="#cli-1" aria-label="Anchor link for: cli-1" style="visibility: hidden;"></a> </h3> <pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">json-to-md</span></span><span class="z-meta z-function-call z-arguments z-shell"> data.json <span class="z-keyword z-operator z-assignment z-redirection z-shell">&gt;</span> out.md <span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> file in, Markdown out</span><span class="z-comment z-line z-number-sign z-shell"> </span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-arguments z-shell">curl<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>s</span> https://api.example.com/items</span> <span class="z-keyword z-operator z-logical z-pipe z-shell">|</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">json-to-md <span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> stdin</span><span class="z-comment z-line z-number-sign z-shell"> </span></span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">json-to-md</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>types</span> data.json <span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> showTypes</span><span class="z-comment z-line z-number-sign z-shell"> </span></span></span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">json-to-md</span></span><span class="z-meta z-function-call z-arguments z-shell"><span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> --</span>json</span> broken.json <span class="z-constant z-numeric z-integer z-decimal z-file-descriptor z-shell">2</span><span class="z-keyword z-operator z-assignment z-redirection z-shell">&gt;</span> error.json <span class="z-comment z-line z-number-sign z-shell"><span class="z-punctuation z-definition z-comment z-begin z-shell">#</span></span><span class="z-comment z-line z-number-sign z-shell"> structured error on stderr</span><span class="z-comment z-line z-number-sign z-shell"> </span></span></span></code></pre> <p>Exit codes: <code>0</code> success, <code>1</code> conversion failed, <code>2</code> usage or I/O error. A failure prints one greppable line:</p> <pre class="z-code"><code><span class="z-text z-plain">json-to-md: DUPLICATE_MEMBER_NAME at 1:13 (first at 1:2): duplicate object member name &quot;name&quot; </span></code></pre> <p>With <code>--json</code> you get the full error object instead:</p> <pre data-lang="json" class="language-json z-code"><code class="language-json" data-lang="json"><span class="z-source z-json"><span class="z-meta z-mapping z-json"><span class="z-punctuation z-section z-mapping z-begin z-json">{</span></span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>code<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span></span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>DUPLICATE_MEMBER_NAME<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span></span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>location<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span></span><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-json"><span class="z-punctuation z-section z-mapping z-begin z-json">{</span></span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>offset<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span></span><span class="z-meta z-mapping z-value z-json"><span class="z-constant z-numeric z-integer z-decimal z-json">12</span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span></span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>line<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span></span><span class="z-meta z-mapping z-value z-json"><span class="z-constant z-numeric z-integer z-decimal z-json">1</span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span></span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>column<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span></span><span class="z-meta z-mapping z-value z-json"><span class="z-constant z-numeric z-integer z-decimal z-json">13</span><span class="z-punctuation z-section z-mapping z-end z-json">}</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span></span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>firstLocation<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span></span><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-json"><span class="z-punctuation z-section z-mapping z-begin z-json">{</span></span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>offset<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span></span><span class="z-meta z-mapping z-value z-json"><span class="z-constant z-numeric z-integer z-decimal z-json">1</span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span></span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>line<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span></span><span class="z-meta z-mapping z-value z-json"><span class="z-constant z-numeric z-integer z-decimal z-json">1</span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span></span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>column<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span></span><span class="z-meta z-mapping z-value z-json"><span class="z-constant z-numeric z-integer z-decimal z-json">2</span><span class="z-punctuation z-section z-mapping z-end z-json">}</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span></span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>message<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span></span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>duplicate object member name <span class="z-constant z-character z-escape z-json">\&quot;</span>name<span class="z-constant z-character z-escape z-json">\&quot;</span><span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-section z-mapping z-end z-json">}</span></span> </span></code></pre> <h2 id="benchmarks">Benchmarks<a class="zola-anchor" href="#benchmarks" aria-label="Anchor link for: benchmarks" style="visibility: hidden;"></a> </h2> <p>I ran everything below on 2026-08-30 on an Apple M3 Pro with 18 GB of RAM, Node 24.14.1, Go 1.23.1, and <code>@rajnandan1/json-to-md</code> 3.0.0 (Go module <code>go/v3.0.0</code>). Every input file was downloaded the same day. The numbers in the project README come from an earlier run on a smaller copy of the Stripe spec; the ones here are fresh.</p> <h3 id="byte-identical-output">Byte-identical output<a class="zola-anchor" href="#byte-identical-output" aria-label="Anchor link for: byte-identical-output" style="visibility: hidden;"></a> </h3> <p>The input is Stripe's OpenAPI spec, <a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/stripe/openapi/blob/master/openapi/spec3.json">spec3.json</a>: 8,028,700 bytes of JSON. Both implementations produced 6,568,122 bytes of Markdown, and the SHA-256 matched:</p> <pre class="z-code"><code><span class="z-text z-plain">dc322742c7f7c00de38da45e8f871fe60cdad67dc67fe8e5eb5d3478cecb7a89 go-out.md </span><span class="z-text z-plain">dc322742c7f7c00de38da45e8f871fe60cdad67dc67fe8e5eb5d3478cecb7a89 ts-out.md </span></code></pre> <p>The corpus promise holds on an 8 MB real-world document as well as on the fixtures.</p> <h3 id="speed">Speed<a class="zola-anchor" href="#speed" aria-label="Anchor link for: speed" style="visibility: hidden;"></a> </h3> <p>Same file, same machine:</p> <table><thead><tr><th>Surface</th><th>Result</th></tr></thead><tbody> <tr><td>Go <code>ConvertText</code></td><td>125 ms per conversion, 64 MB/s (<code>go test -bench</code>, 10 iterations)</td></tr> <tr><td>TypeScript <code>convertJsonText</code>, Node 24</td><td>207 ms median over 11 runs (min 185, max 240)</td></tr> <tr><td>CLI, whole process, <code>spec3.json</code></td><td>0.13 s wall clock, consistent across 10 runs</td></tr> <tr><td>CLI, whole process, a 6 KB GitHub API response</td><td>under 10 ms</td></tr> </tbody></table> <p>Go runs about 1.7x faster than Node on this document. Both are fast enough that conversion will never be the slow step in a pipeline that also makes a network call.</p> <p>Reproduce with any large JSON file:</p> <pre data-lang="sh" class="language-sh z-code"><code class="language-sh" data-lang="sh"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">git</span></span><span class="z-meta z-function-call z-arguments z-shell"> clone https://github.com/rajnandan1/json-to-md</span> <span class="z-keyword z-operator z-logical z-and z-shell">&amp;&amp;</span> <span class="z-meta z-function-call z-shell"><span class="z-support z-function z-cd z-shell">cd</span></span><span class="z-meta z-function-call z-arguments z-shell"> json-to-md</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">pnpm</span></span><span class="z-meta z-function-call z-arguments z-shell"> install</span> <span class="z-keyword z-operator z-logical z-and z-shell">&amp;&amp;</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">pnpm</span></span><span class="z-meta z-function-call z-arguments z-shell"> build</span> <span class="z-keyword z-operator z-logical z-and z-shell">&amp;&amp;</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">node</span></span><span class="z-meta z-function-call z-arguments z-shell"> scripts/bench-file.mjs path/to/big.json 11</span> </span><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-support z-function z-cd z-shell">cd</span></span><span class="z-meta z-function-call z-arguments z-shell"> go</span> <span class="z-keyword z-operator z-logical z-and z-shell">&amp;&amp;</span> <span class="z-variable z-other z-readwrite z-assignment z-shell">BENCH_FILE</span><span class="z-keyword z-operator z-assignment z-shell">=</span><span class="z-string z-unquoted z-shell">path/to/big.json</span> <span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">go</span></span><span class="z-meta z-function-call z-arguments z-shell"> test<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>bench</span> ConvertTextFile<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>benchtime</span> 10x<span class="z-variable z-parameter z-option z-shell"><span class="z-punctuation z-definition z-parameter z-shell"> -</span>run</span> <span class="z-string z-quoted z-single z-shell"><span class="z-punctuation z-definition z-string z-begin z-shell">&#39;</span>^$<span class="z-punctuation z-definition z-string z-end z-shell">&#39;</span></span></span> </span></code></pre> <h3 id="tokens-where-markdown-wins-and-where-it-loses">Tokens: where Markdown wins and where it loses<a class="zola-anchor" href="#tokens-where-markdown-wins-and-where-it-loses" aria-label="Anchor link for: tokens-where-markdown-wins-and-where-it-loses" style="visibility: hidden;"></a> </h3> <p>The pitch for Markdown in prompts is that models pay for JSON punctuation. I wanted a number from my own files instead of a quoted one, so I counted tokens with tiktoken's <code>o200k_base</code> encoding (the GPT-4o and GPT-5 tokenizer) for each document in three forms: pretty-printed JSON with two-space indentation, minified JSON, and json-to-md output. <code>cl100k_base</code> gave the same direction on every file, within a few points.</p> <table><thead><tr><th>Document</th><th>Shape</th><th>Pretty JSON</th><th>Minified JSON</th><th>Markdown</th><th>vs pretty</th><th>vs minified</th></tr></thead><tbody> <tr><td><a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/vega/vega-datasets/blob/main/data/cars.json">cars.json</a></td><td>406 flat records, 9 columns</td><td>36,106</td><td>23,575</td><td>15,414</td><td>-57%</td><td>-35%</td></tr> <tr><td><a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/vega/vega-datasets/blob/main/data/movies.json">movies.json</a></td><td>3,201 flat records, 16 columns</td><td>500,615</td><td>343,404</td><td>217,508</td><td>-57%</td><td>-37%</td></tr> <tr><td>JSONPlaceholder <code>/comments</code></td><td>500 flat records</td><td>45,260</td><td>35,761</td><td>32,499</td><td>-28%</td><td>-9%</td></tr> <tr><td>JSONPlaceholder <code>/posts</code></td><td>100 flat records</td><td>7,690</td><td>6,091</td><td>5,659</td><td>-26%</td><td>-7%</td></tr> <tr><td>JSONPlaceholder <code>/users</code></td><td>10 records, two nested objects each</td><td>1,829</td><td>1,223</td><td>1,503</td><td>-18%</td><td>+23%</td></tr> <tr><td>PyPI <code>requests</code> metadata</td><td>deep object, release history</td><td>97,138</td><td>82,258</td><td>119,833</td><td>+23%</td><td>+46%</td></tr> <tr><td>Stripe <code>spec3.json</code></td><td>deep object, 8 MB</td><td>1,371,581</td><td>902,685</td><td>1,729,974</td><td>+26%</td><td>+92%</td></tr> <tr><td>npm registry doc for this package</td><td>deep object</td><td>7,971</td><td>7,105</td><td>10,858</td><td>+36%</td><td>+53%</td></tr> <tr><td>GitHub API, 30 VS Code issues</td><td>nested records, URL heavy</td><td>64,952</td><td>55,638</td><td>88,931</td><td>+37%</td><td>+60%</td></tr> <tr><td>GitHub API, 30 commits</td><td>nested records, URL heavy</td><td>49,892</td><td>43,335</td><td>70,638</td><td>+42%</td><td>+63%</td></tr> <tr><td>GitHub API, one repo</td><td>one object, 49 URL fields</td><td>1,848</td><td>1,556</td><td>2,846</td><td>+54%</td><td>+83%</td></tr> </tbody></table> <p>Two shapes, two answers.</p> <p><strong>Flat arrays of records win, and win big.</strong> The two Vega datasets drop 57% against pretty JSON and 35% to 37% against minified. The header row spells each key once; JSON spells it once per record. If you are feeding query results, CSV-shaped exports, or log rows to a model, this is the case the tool was built for.</p> <p><strong>Nested API responses lose.</strong> Every GitHub, npm, PyPI, and Stripe document cost more as Markdown than as JSON, by 23% to 54% against pretty-printed and 46% to 92% against minified. The README cites a 16% saving on one document. On my files the answer depends on shape, and I would rather publish the spread than the best case.</p> <p>I traced where the extra tokens go on the GitHub commit list, which is the worst realistic case:</p> <table><thead><tr><th>Step</th><th>Tokens</th></tr></thead><tbody> <tr><td>Minified JSON</td><td>43,335</td></tr> <tr><td>Pretty JSON</td><td>49,892</td></tr> <tr><td>Markdown as generated</td><td>70,638</td></tr> <tr><td>Markdown with URL links collapsed to bare URLs</td><td>52,814</td></tr> <tr><td>...and with backslash escapes removed</td><td>50,136</td></tr> </tbody></table> <p>The URL rule is the biggest cost. Any string that is a whole <code>http(s)</code> URL renders as <code>[url](url)</code>, which spells the URL twice. That commit list has 770 URL-valued fields, and the doubling alone accounts for 17,824 tokens, a quarter of the output. Escaping is second: GitHub's keys are snake_case, and each <code>_</code> becomes <code>\_</code>, which splits a key like <code>node_id</code> into more tokens than the bare word. 1,369 escape sequences, 1,324 of them underscores, cost another 2,678 tokens. Take both away and the Markdown lands within 1% of pretty-printed JSON and 16% above minified; that remainder is the heading structure, where every leaf under a nested object gets a <code>###</code> line and two blank lines.</p> <p>The Stripe spec has almost no URLs and still loses, because it is objects nested in objects nested in objects. Its Markdown carries 127,117 escape sequences, 80,733 of them <code>\_</code>, and even with all of them removed it sits at 1,540,277 tokens against 902,685 minified. Deep object trees are the shape this projection handles worst for token count, even though they are the shape it handles best for reading.</p> <p>So my rule for prompts:</p> <ul> <li>Tabular data, log rows, anything a person will also read: convert it.</li> <li>Deeply nested, URL-dense API payloads that only the model will read: keep them as minified JSON.</li> </ul> <p>An option to render URLs as bare text would recover most of the gap on API responses. It is not in 3.0.0.</p> <h2 id="try-it">Try it<a class="zola-anchor" href="#try-it" aria-label="Anchor link for: try-it" style="visibility: hidden;"></a> </h2> <p>The <a rel="noopener nofollow noreferrer" target="_blank" href="https://json-to-md.rajnandan.com/demo/">playground</a> loads the released library from the CDN. Paste JSON, switch between the rendered preview and the raw Markdown, and flip between the text and value entry points to watch numeric spelling survive or not.</p> <p>Source, corpus, and CI are at <a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/rajnandan1/json-to-md">github.com/rajnandan1/json-to-md</a>. MIT licensed.</p> 2026-08-30T00:00:00+00:00 2026-08-30T00:00:00+00:00 https://rajnandan.com/posts/introducing-ken/ <p>I built <a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/rajnandan1/ken">ken</a> because the code my coding agent writes on Monday is the code I maintain in December. Agents are fast. Speed is not the problem. The problem is that an agent under a bug ticket will add a fourth patch to a function that already carries three, reach for a web framework to serve two JSON routes, and guard the call site the ticket named while the shared helper underneath stays broken. Every one of those moves passes the acceptance test. Every one of them is something I pay for later.</p> <p>ken is a ruleset that loads into the agent at session start. The rules come from one programmer. In the summer of 1969, with his wife away for a month, Ken Thompson wrote the Unix kernel, a shell, an editor, and an assembler, about a week each. Forty years later he sat with Rob Pike and Robert Griesemer and designed Go, where, in his words, "all three of us had to be talked into every feature in the language." Between those two points he said enough about how he works that you can write it down as a procedure. ken is that procedure, with a source for each line.</p> <p>Install it in Claude Code:</p> <pre class="z-code"><code><span class="z-text z-plain">/plugin marketplace add rajnandan1/ken </span><span class="z-text z-plain">/plugin install ken@ken </span></code></pre> <p>Cursor, Windsurf, Cline, Copilot, Gemini CLI, OpenCode, pi, Hermes, Qoder, Kiro, Grok, Codex, and any agent that reads <code>AGENTS.md</code> get the same rules through thin adapters; <a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/rajnandan1/ken/blob/main/docs/agent-portability.md">docs/agent-portability.md</a> lists each one. MCP-only hosts get an MCP server. The hooks need Node on your PATH.</p> <h2 id="the-loop">The loop<a class="zola-anchor" href="#the-loop" aria-label="Anchor link for: the-loop" style="visibility: hidden;"></a> </h2> <p>ken runs six steps, in order, on every task:</p> <ol> <li><strong>Think first.</strong> Build the mental model before touching the code. Pike's account of pairing with Thompson at Bell Labs: Ken would stand and think, then name the bug before Pike had finished looking at the code.</li> <li><strong>Steal, don't invent.</strong> A proven idea from this codebase, the stdlib, or a classic beats a new one. Thompson on Unix: "the things we stole: we stole a shell out of MULTICS." Pipes took years of paring McIlroy's idea down and then went in overnight.</li> <li><strong>Build bottom-up</strong> from primitives you can explain line by line. "When I see a top-down description of a system or language that has infinite libraries described by layers and layers, all I just see is a morass."</li> <li><strong>When in doubt, use brute force.</strong> The plain loop and the flat array until a measurement says otherwise.</li> <li><strong>Try it.</strong> "Try it, and if it doesn't work, throw it out and do it again."</li> <li><strong>Throw it out when it fights you.</strong> "Code by itself almost rots and it's gotta be rewritten." Before fixing a bug, count the unit's fix-comment trail. Three or more prior fixes means the unit gets rewritten. Nobody adds entry four.</li> </ol> <p>Under the loop sit the rules the loop rests on. Features default to no. Interfaces few and small, because open, close, read, and write ran a whole operating system. No layer that only translates ("layer after layer after layer that does nothing except translate" is the line that scares him about modern programming). Minimal trusted base: read enough of a dependency to vouch for it, or write the few lines yourself. Know every line of your own diff before you call it done. Debug the model, not the symptom. No ceremony.</p> <p>Each rule maps to a quote in <a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/rajnandan1/ken/blob/main/PROVENANCE.md">PROVENANCE.md</a>, rated PRIMARY (his words, verified against the source), WITNESS (a colleague's first-hand account), or ATTRIBUTED. "When in doubt, use brute force" is ATTRIBUTED: the earliest print is Bentley's 1985 column, and no researcher has found Thompson saying it. The famous "one of my most productive days was throwing away 1,000 lines of code" has no located source at all, and ken says so rather than putting it in his mouth.</p> <p>When the agent takes a deliberate shortcut, it marks the ceiling in the code:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> ken: linear scan; sort + bisect when n &gt; 10k measured</span> </span></code></pre> <p><code>/ken-debt</code> later harvests every such marker into a ledger, so a deferral cannot turn into a permanent decision without anyone noticing.</p> <h2 id="with-it-and-without-it">With it and without it<a class="zola-anchor" href="#with-it-and-without-it" aria-label="Anchor link for: with-it-and-without-it" style="visibility: hidden;"></a> </h2> <p>Rules in a README prove nothing, so I ran the same tickets through headless Claude Code sessions with ken loaded and without it. The codebase is <code>ledgerd</code>, the small Python invoicing app ken's own benchmark seeds: eleven modules, stdlib only, with planted traps. Each cell is a fresh copy of the seed, one ticket, one <code>claude -p</code> session on claude-haiku-4-5 with Bash disabled so the agent can only read and write files. Three runs per arm per ticket, 24 sessions, $8.19 on the API. I scored the files each session left behind, never the prose.</p> <h3 id="ticket-1-a-function-on-its-third-patch">Ticket 1: a function on its third patch<a class="zola-anchor" href="#ticket-1-a-function-on-its-third-patch" aria-label="Anchor link for: ticket-1-a-function-on-its-third-patch" style="visibility: hidden;"></a> </h3> <p><code>duration.py</code> parses <code>1h30m45s</code> into seconds and carries three dated fix comments. The ticket: a bare number like <code>'45'</code> returns 0, "this function has already been patched three times, fix it without breaking the previous cases."</p> <p>Without ken, all three sessions did the same thing:</p> <pre data-lang="diff" class="language-diff z-code"><code class="language-diff" data-lang="diff"><span class="z-source z-diff"> elif u == &quot;s&quot;: </span><span class="z-source z-diff"> total += float(num or 0) </span><span class="z-source z-diff"> num = &quot;&quot; </span><span class="z-source z-diff"><span class="z-markup z-inserted z-diff"><span class="z-punctuation z-definition z-inserted z-diff">+</span> # fix 2026-08-30 (#XX): bare numbers (no unit) are seconds from tracker exports </span></span><span class="z-source z-diff"><span class="z-markup z-inserted z-diff"><span class="z-punctuation z-definition z-inserted z-diff">+</span> if num: </span></span><span class="z-source z-diff"><span class="z-markup z-inserted z-diff"><span class="z-punctuation z-definition z-inserted z-diff">+</span> total += float(num) </span></span><span class="z-source z-diff"> return int(total) </span></code></pre> <p>Patch four, with its own dated comment, ready for patch five. The tests pass. The trail is now four entries long.</p> <p>With ken, two of three sessions took the trail out and restructured the unit:</p> <pre data-lang="diff" class="language-diff z-code"><code class="language-diff" data-lang="diff"><span class="z-source z-diff"> def parse_duration(s): </span><span class="z-source z-diff"><span class="z-markup z-deleted z-diff"><span class="z-punctuation z-definition z-deleted z-diff">-</span> &quot;&quot;&quot;Parse &#39;1h30m45s&#39; into total seconds.&quot;&quot;&quot; </span></span><span class="z-source z-diff"><span class="z-markup z-deleted z-diff"><span class="z-punctuation z-definition z-deleted z-diff">-</span> # fix 2026-03-11 (#41): imports sometimes hand us None for the duration cell </span></span><span class="z-source z-diff"><span class="z-markup z-inserted z-diff"><span class="z-punctuation z-definition z-inserted z-diff">+</span> &quot;&quot;&quot;Parse &#39;1h30m45s&#39; into total seconds. Bare numbers default to seconds.&quot;&quot;&quot; </span></span><span class="z-source z-diff"> if s is None: </span><span class="z-source z-diff"> return 0 </span><span class="z-source z-diff"> total = 0.0 </span><span class="z-source z-diff">@@ </span><span class="z-source z-diff"> if ch.isdigit() or ch == &quot;.&quot;: </span><span class="z-source z-diff"> num += ch </span><span class="z-source z-diff"> else: </span><span class="z-source z-diff"><span class="z-markup z-deleted z-diff"><span class="z-punctuation z-definition z-deleted z-diff">-</span> # fix 2026-05-02 (#58): tolerate uppercase unit letters from the web form </span></span><span class="z-source z-diff"><span class="z-markup z-deleted z-diff"><span class="z-punctuation z-definition z-deleted z-diff">-</span> u = ch.lower() </span></span><span class="z-source z-diff"><span class="z-markup z-deleted z-diff"><span class="z-punctuation z-definition z-deleted z-diff">-</span> if u == &quot;h&quot;: </span></span><span class="z-source z-diff"><span class="z-markup z-deleted z-diff"><span class="z-punctuation z-definition z-deleted z-diff">-</span> # fix 2026-07-19 (#77): float amounts like &#39;1.5h&#39; from the tracker export </span></span><span class="z-source z-diff"><span class="z-markup z-deleted z-diff"><span class="z-punctuation z-definition z-deleted z-diff">-</span> total += float(num) * 3600 </span></span><span class="z-source z-diff"><span class="z-markup z-deleted z-diff"><span class="z-punctuation z-definition z-deleted z-diff">-</span> elif u == &quot;m&quot;: </span></span><span class="z-source z-diff"><span class="z-markup z-deleted z-diff"><span class="z-punctuation z-definition z-deleted z-diff">-</span> total += float(num) * 60 </span></span><span class="z-source z-diff"><span class="z-markup z-deleted z-diff"><span class="z-punctuation z-definition z-deleted z-diff">-</span> elif u == &quot;s&quot;: </span></span><span class="z-source z-diff"><span class="z-markup z-deleted z-diff"><span class="z-punctuation z-definition z-deleted z-diff">-</span> total += float(num or 0) </span></span><span class="z-source z-diff"><span class="z-markup z-deleted z-diff"><span class="z-punctuation z-definition z-deleted z-diff">-</span> num = &quot;&quot; </span></span><span class="z-source z-diff"><span class="z-markup z-inserted z-diff"><span class="z-punctuation z-definition z-inserted z-diff">+</span> if num: </span></span><span class="z-source z-diff"><span class="z-markup z-inserted z-diff"><span class="z-punctuation z-definition z-inserted z-diff">+</span> unit = ch.lower() </span></span><span class="z-source z-diff"><span class="z-markup z-inserted z-diff"><span class="z-punctuation z-definition z-inserted z-diff">+</span> amount = float(num) </span></span><span class="z-source z-diff"><span class="z-markup z-inserted z-diff"><span class="z-punctuation z-definition z-inserted z-diff">+</span> if unit == &quot;h&quot;: </span></span><span class="z-source z-diff"><span class="z-markup z-inserted z-diff"><span class="z-punctuation z-definition z-inserted z-diff">+</span> total += amount * 3600 </span></span><span class="z-source z-diff"><span class="z-markup z-inserted z-diff"><span class="z-punctuation z-definition z-inserted z-diff">+</span> elif unit == &quot;m&quot;: </span></span><span class="z-source z-diff"><span class="z-markup z-inserted z-diff"><span class="z-punctuation z-definition z-inserted z-diff">+</span> total += amount * 60 </span></span><span class="z-source z-diff"><span class="z-markup z-inserted z-diff"><span class="z-punctuation z-definition z-inserted z-diff">+</span> elif unit == &quot;s&quot;: </span></span><span class="z-source z-diff"><span class="z-markup z-inserted z-diff"><span class="z-punctuation z-definition z-inserted z-diff">+</span> total += amount </span></span><span class="z-source z-diff"><span class="z-markup z-inserted z-diff"><span class="z-punctuation z-definition z-inserted z-diff">+</span> num = &quot;&quot; </span></span><span class="z-source z-diff"><span class="z-markup z-inserted z-diff"><span class="z-punctuation z-definition z-inserted z-diff">+</span> if num: </span></span><span class="z-source z-diff"><span class="z-markup z-inserted z-diff"><span class="z-punctuation z-definition z-inserted z-diff">+</span> total += float(num) </span></span><span class="z-source z-diff"> return int(total) </span></code></pre> <p>The session's closing line: <code>→ rewrote: parse_duration, threw away: three-patch fix trail, ceiling: none; linear scan handles all formats.</code> Bare numbers now hold as a documented default in the docstring, and each earlier fix survives as behavior rather than as a comment explaining a hack. Both the visible tests and my hidden tests (<code>'90.5'</code>, <code>'10m30'</code>) pass on every run of both arms, so the rewrite cost nothing in correctness.</p> <p>The third ken session restructured the same loop but left the three comments in place while telling me "per ken discipline, when a unit reaches three patches, rewriting is cleaner than stacking a fourth fix." It said the words and skipped the deletion. I count that as a miss, and ken's own scorer counts it the same way: the trail is gone or it is not.</p> <p>Two things to be plain about. First, this is a restructure of the same twenty lines, closer to Thompson's "walk through it line by line" than to a blank-page rewrite. Second, the ticket itself said "patched three times." Earlier benchmark rounds showed that on this model the rule fires when the evidence sits at the edit site, and a comment trail is that evidence. I did not test a rotted unit whose history lived only in <code>git log</code>.</p> <h3 id="ticket-2-two-json-routes">Ticket 2: two JSON routes<a class="zola-anchor" href="#ticket-2-two-json-routes" aria-label="Anchor link for: ticket-2-two-json-routes" style="visibility: hidden;"></a> </h3> <p>"Add an HTTP JSON API for ledgerd's accounts in a new <code>api.py</code>. <code>python api.py --port 8123</code> must start it. GET /balances returns the balances. POST /deposit deposits and returns the new balance." Nothing in the prompt names a framework or a module. ledgerd has no <code>requirements.txt</code>.</p> <p>Without ken, two of three sessions wrote this:</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">argparse</span></span></span> </span><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">json</span></span></span> </span><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-from z-python">from</span></span><span class="z-meta z-statement z-import z-python"><span class="z-meta z-import-source z-python"> <span class="z-meta z-import-path z-python"><span class="z-meta z-import-name z-python">flask</span></span> <span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span></span></span></span><span class="z-meta z-statement z-import z-python"></span><span class="z-meta z-statement z-import z-python"> <span class="z-meta z-generic-name z-python">Flask</span><span class="z-punctuation z-separator z-import-list z-python">,</span> <span class="z-meta z-generic-name z-python">request</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">accounts</span></span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">app</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">Flask</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-support z-variable z-magic z-python">__name__</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-annotation z-python"><span class="z-punctuation z-definition z-annotation z-python">@</span><span class="z-meta z-annotation z-function z-python"></span></span><span class="z-meta z-annotation z-function z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">app</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-annotation z-function z-python"><span class="z-meta z-generic-name z-python">route</span></span></span></span><span class="z-meta z-annotation z-function z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span></span><span class="z-meta z-annotation z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">/balances<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-variable z-parameter z-python">methods</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-sequence z-list z-python"><span class="z-punctuation z-section z-sequence z-begin z-python">[</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">GET<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-section z-sequence z-end z-python">]</span></span></span><span class="z-meta z-annotation z-function z-python"><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"><span class="z-meta z-function z-python"><span class="z-storage z-type z-function z-python"><span class="z-keyword z-declaration z-function z-python">def</span></span> <span class="z-entity z-name z-function z-python"><span class="z-meta z-generic-name z-python">get_balances</span></span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-begin z-python">(</span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-end z-python">)</span></span><span class="z-meta z-function z-python"><span class="z-punctuation z-section z-function z-begin z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-keyword z-control z-flow z-return z-python">return</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">accounts</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">balances</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-annotation z-python"><span class="z-punctuation z-definition z-annotation z-python">@</span><span class="z-meta z-annotation z-function z-python"></span></span><span class="z-meta z-annotation z-function z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">app</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-annotation z-function z-python"><span class="z-meta z-generic-name z-python">route</span></span></span></span><span class="z-meta z-annotation z-function z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span></span><span class="z-meta z-annotation z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">/deposit<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-variable z-parameter z-python">methods</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-sequence z-list z-python"><span class="z-punctuation z-section z-sequence z-begin z-python">[</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">POST<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-section z-sequence z-end z-python">]</span></span></span><span class="z-meta z-annotation z-function z-python"><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"><span class="z-meta z-function z-python"><span class="z-storage z-type z-function z-python"><span class="z-keyword z-declaration z-function z-python">def</span></span> <span class="z-entity z-name z-function z-python"><span class="z-meta z-generic-name z-python">post_deposit</span></span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-begin z-python">(</span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-end z-python">)</span></span><span class="z-meta z-function z-python"><span class="z-punctuation z-section z-function z-begin z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">data</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">request</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">get_json</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">accounts</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">deposit</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-item-access z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">data</span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-begin z-python">[</span></span><span class="z-meta z-item-access z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">acct<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-end z-python">]</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-item-access z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">data</span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-begin z-python">[</span></span><span class="z-meta z-item-access z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">cents<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-end z-python">]</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-keyword z-control z-flow z-return z-python">return</span> <span class="z-meta z-mapping z-python"><span class="z-punctuation z-section z-mapping z-begin z-python">{</span></span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">balance<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-item-access z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">accounts</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">balances</span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-begin z-python">[</span></span><span class="z-meta z-item-access z-arguments z-python"><span class="z-meta z-item-access z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">data</span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-begin z-python">[</span></span><span class="z-meta z-item-access z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">acct<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-end z-python">]</span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-end z-python">]</span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-section z-mapping z-end z-python">}</span></span> </span></code></pre> <p>Twenty lines and one dependency that no file in the project had asked for. It passed my acceptance test on my Mac, and it took me a minute to work out why: the Xcode-bundled Python happens to ship Flask 3.1. On any other host <code>api.py</code> dies on line 3.</p> <p>With ken, three of three sessions served the routes on <code>http.server</code>:</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-from z-python">from</span></span><span class="z-meta z-statement z-import z-python"><span class="z-meta z-import-source z-python"> <span class="z-meta z-import-path z-python"><span class="z-meta z-import-name z-python">http</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">server</span></span> <span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span></span></span></span><span class="z-meta z-statement z-import z-python"></span><span class="z-meta z-statement z-import z-python"> <span class="z-meta z-generic-name z-python">HTTPServer</span><span class="z-punctuation z-separator z-import-list z-python">,</span> <span class="z-meta z-generic-name z-python">BaseHTTPRequestHandler</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">accounts</span></span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-class z-python"><span class="z-storage z-type z-class z-python"><span class="z-keyword z-declaration z-class z-python">class</span></span> <span class="z-entity z-name z-class z-python"><span class="z-meta z-generic-name z-python">Handler</span></span><span class="z-meta z-class z-inheritance z-python"><span class="z-punctuation z-section z-inheritance z-begin z-python">(</span></span></span><span class="z-meta z-class z-inheritance z-python"><span class="z-entity z-other z-inherited-class z-python">BaseHTTPRequestHandler</span><span class="z-punctuation z-section z-inheritance z-end z-python">)</span></span><span class="z-meta z-class z-python"><span class="z-punctuation z-section z-class z-begin z-python">:</span></span> </span><span class="z-source z-python"><span class="z-meta z-function z-python"> <span class="z-storage z-type z-function z-python"><span class="z-keyword z-declaration z-function z-python">def</span></span> <span class="z-entity z-name z-function z-python"><span class="z-meta z-generic-name z-python">do_GET</span></span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-begin z-python">(</span></span><span class="z-meta z-function z-parameters z-python"><span class="z-variable z-parameter z-python">self</span><span class="z-punctuation z-section z-parameters z-end z-python">)</span></span><span class="z-meta z-function z-python"><span class="z-punctuation z-section z-function z-begin z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-statement z-conditional z-if z-python"><span class="z-keyword z-control z-conditional z-if z-python">if</span> <span class="z-meta z-qualified-name z-python"><span class="z-variable z-language z-python">self</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">path</span></span> <span class="z-keyword z-operator z-comparison z-python">==</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">/balances<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-block z-conditional z-if z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-language z-python">self</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">send_response</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-constant z-numeric z-integer z-decimal z-python">200</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-language z-python">self</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">send_header</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">Content-Type<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">application/json<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-language z-python">self</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">end_headers</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-language z-python">self</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">wfile</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">write</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">json</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">dumps</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">accounts</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">balances</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">encode</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-meta z-statement z-conditional z-else z-python"><span class="z-keyword z-control z-conditional z-else z-python">else</span><span class="z-punctuation z-section z-block z-conditional z-else z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-language z-python">self</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">send_response</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-constant z-numeric z-integer z-decimal z-python">404</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-language z-python">self</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">end_headers</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-function z-python"> <span class="z-storage z-type z-function z-python"><span class="z-keyword z-declaration z-function z-python">def</span></span> <span class="z-entity z-name z-function z-python"><span class="z-meta z-generic-name z-python">do_POST</span></span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-begin z-python">(</span></span><span class="z-meta z-function z-parameters z-python"><span class="z-variable z-parameter z-python">self</span><span class="z-punctuation z-section z-parameters z-end z-python">)</span></span><span class="z-meta z-function z-python"><span class="z-punctuation z-section z-function z-begin z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-statement z-conditional z-if z-python"><span class="z-keyword z-control z-conditional z-if z-python">if</span> <span class="z-meta z-qualified-name z-python"><span class="z-variable z-language z-python">self</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">path</span></span> <span class="z-keyword z-operator z-comparison z-python">==</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">/deposit<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-block z-conditional z-if z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">length</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-type z-python">int</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-language z-python">self</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">headers</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">get</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">Content-Length<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-constant z-numeric z-integer z-decimal z-python">0</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">body</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">json</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">loads</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-language z-python">self</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">rfile</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">read</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">length</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">decode</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">accounts</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">deposit</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-item-access z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">body</span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-begin z-python">[</span></span><span class="z-meta z-item-access z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">acct<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-end z-python">]</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-item-access z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">body</span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-begin z-python">[</span></span><span class="z-meta z-item-access z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">cents<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-end z-python">]</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-constant z-language z-python">...</span> </span></code></pre> <p>Closing line: <code>→ stolen: stdlib http.server, json, ceiling: sync handlers.</code> Thirty-seven to forty-two lines against twenty. ken wrote twice the code and added nothing to the trusted base, and it named the ceiling (a synchronous server) instead of hiding it. That trade is the whole point. Lines of code measure typing. What you maintain is the set of things that can break, and Flask is one more of them.</p> <h3 id="ticket-3-the-bug-that-lives-one-function-down">Ticket 3: the bug that lives one function down<a class="zola-anchor" href="#ticket-3-the-bug-that-lives-one-function-down" aria-label="Anchor link for: ticket-3-the-bug-that-lives-one-function-down" style="visibility: hidden;"></a> </h3> <p><code>accounts.py</code> has <code>_debit</code>, called by both <code>transfer</code> and <code>withdraw</code>. Neither checks the balance. The ticket reports the symptom through <code>transfer</code>: "after some transfers an account ended up with a negative balance; reject with <code>ValueError('insufficient funds')</code>."</p> <p>All six sessions, ken or not, wrote the identical two lines:</p> <pre data-lang="diff" class="language-diff z-code"><code class="language-diff" data-lang="diff"><span class="z-source z-diff"> def transfer(src, dst, cents): </span><span class="z-source z-diff"> &quot;&quot;&quot;Move `cents` from src to dst.&quot;&quot;&quot; </span><span class="z-source z-diff"><span class="z-markup z-inserted z-diff"><span class="z-punctuation z-definition z-inserted z-diff">+</span> if balances.get(src, 0) &lt; cents: </span></span><span class="z-source z-diff"><span class="z-markup z-inserted z-diff"><span class="z-punctuation z-definition z-inserted z-diff">+</span> raise ValueError(&#39;insufficient funds&#39;) </span></span><span class="z-source z-diff"> _debit(src, cents) </span><span class="z-source z-diff"> deposit(dst, cents) </span></code></pre> <p><code>_debit</code> untouched. <code>withdraw("carol", 500)</code> on a balance of 100 still goes through and leaves carol at -400. My hidden test caught it in six of six runs.</p> <p>ken has a rule for this. "Debug the model, not the symptom: before editing a function to fix a bug, list its callers and callees; if the flaw lives in a shared helper, fix the helper and name the sibling callers." The rule did not fire once. The <a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/rajnandan1/ken/tree/main/benchmarks/results">results directory</a> already records this: I tried three wordings of that rule across three measured rounds and reverted each one, because on a single-shot haiku session the agent skims toward the named site and never opens the caller list. My working explanation is that rules fire when the evidence is visible where the edit happens. A fix-comment trail is visible there. A sibling caller in another function is not. In an interactive session, with the agent reading more of the file and you asking "who else calls this?", the rule has fired in field reports. Headless and single-shot, it does not.</p> <h3 id="ticket-4-the-helper-a-few-files-over">Ticket 4: the helper a few files over<a class="zola-anchor" href="#ticket-4-the-helper-a-few-files-over" aria-label="Anchor link for: ticket-4-the-helper-a-few-files-over" style="visibility: hidden;"></a> </h3> <p>"Add <code>export_filename(inv_id)</code> returning <code>&lt;slug&gt;.csv</code>. Slugs must stay consistent with how the rest of the project builds them." <code>textutils.slugify</code> exists and transliterates accents. Six of six sessions imported it, and my hidden test (<code>Café Olé</code> becomes <code>cafe-ole.csv</code>) passed on every run. On a codebase this size, both arms find the helper. The reuse rule earns its keep on repos where the helper is not two files away, and I have no measurement of that here.</p> <h3 id="the-tally">The tally<a class="zola-anchor" href="#the-tally" aria-label="Anchor link for: the-tally" style="visibility: hidden;"></a> </h3> <table><thead><tr><th>Ticket</th><th>Without ken</th><th>With ken</th></tr></thead><tbody> <tr><td>Rotted unit: trail removed, tests pass</td><td>0 of 3</td><td>2 of 3</td></tr> <tr><td>API: new third-party import</td><td>2 of 3</td><td>0 of 3</td></tr> <tr><td>Shared helper fixed, sibling caller safe</td><td>0 of 3</td><td>0 of 3</td></tr> <tr><td>Project helper reused</td><td>3 of 3</td><td>3 of 3</td></tr> </tbody></table> <p>Cost and time went both ways. The rewrite sessions ran faster with ken (191 seconds median against 372) at about the same cost. The API sessions ran slower and cost more ($0.57 median against $0.06), which is what you'd expect when one arm writes a request handler by hand and the other writes six lines around a framework. ken does not make sessions cheaper. It changes what the session leaves in the repository.</p> <h2 id="the-rest-of-the-plugin">The rest of the plugin<a class="zola-anchor" href="#the-rest-of-the-plugin" aria-label="Anchor link for: the-rest-of-the-plugin" style="visibility: hidden;"></a> </h2> <p><code>/ken-review</code> reads a diff for method violations only, one line each, and stays out of correctness:</p> <pre class="z-code"><code><span class="z-text z-plain">L12-60: rot: third patch on this cache&#39;s invalidation. Rewrite on a plain dict + mtime check, ~25 lines. </span><span class="z-text z-plain">L4: unvouched: left-pad-like microdep imported unread. 3 lines inline, trusted base shrinks by one. </span><span class="z-text z-plain">svc.py:L88: layer: OrderManager delegates every call to OrderRepo. Delete, callers hit OrderRepo. </span><span class="z-text z-plain">L30-52: fancy: hand-rolled B-tree for 40 entries. Flat array + linear scan; ken: revisit when n &gt; 10k measured. </span><span class="z-text z-plain">L71: ceremony: config flag nobody sets gates one constant. Inline the constant. </span><span class="z-text z-plain">net: -140 lines, -1 layer, trusted base -1 dep possible. </span></code></pre> <p><code>/ken-audit</code> runs the same hunt across the whole tree, weighted by fix-commit density (<code>git log --format= --name-only | sort | uniq -c | sort -rn</code>), because rot lives where patches pile up. <code>/ken-debt</code> is the ceilings ledger. <code>/ken-gain</code> shows the benchmark medians as ASCII bars and refuses to print a per-repo savings number, on the grounds that the unbuilt version was never written, so there is no baseline to subtract from.</p> <p>Three intensities. <code>/ken lite</code> builds what you asked and names the Thompson move in one line. <code>/ken</code> (full, the default) enforces the loop. <code>/ken ultra</code> rewrites first on any rotted unit, makes you argue for each feature, and freezes the trusted base. <code>/ken default ultra</code> persists a level across sessions; <code>stop ken</code> turns it off. A statusline badge shows <code>[KEN]</code> or <code>[KEN:ULTRA]</code> so you know which agent you are talking to.</p> <p>The safety carve-outs are explicit. Brute force stops at input validation on trust boundaries, error handling that prevents data loss, security, accessibility, and whatever you asked for. The agent traces a unit end to end before it calls it rot, and a throwaway round still leaves one runnable check behind.</p> <h2 id="how-ken-measures-itself">How ken measures itself<a class="zola-anchor" href="#how-ken-measures-itself" aria-label="Anchor link for: how-ken-measures-itself" style="visibility: hidden;"></a> </h2> <p>The part of this project I am proudest of is the results directory. ken's maintenance benchmark runs a sequence of tickets against one persistent workspace, commits after each, and scores what survives at the end plus deterministic rates: reuse, root cause, rewrite-on-rot, and now the trusted base. No LLM judges. Before any API spend, a scripted "good" agent and a scripted lazy twin go through the whole pipeline; the twin passes every visible test and must be caught by every rate, or the run refuses to start.</p> <p>The first measured round, 26 August, showed no ken advantage. Rewrite 0 of 2 on both arms, root cause pointing against ken. I published it, and the sonnet round that replicated the null the same day. Iteration 1 changed the rewrite rule from an aspiration ("code rots") into a procedure ("count the fix-comment trail; three or more means rewrite") and rewrite went from 0 of 6 rot cells to 6 of 6. That change shipped as v1.1 under a verdict rule written down before the data came in. Iterations 2, 3, and 4 each tried a wording and each was reverted, with the reason recorded. Most projects publish the round that worked. ken's history has one KEEP and three REVERTs, in the open.</p> <h2 id="what-i-changed-before-publishing-this">What I changed before publishing this<a class="zola-anchor" href="#what-i-changed-before-publishing-this" aria-label="Anchor link for: what-i-changed-before-publishing-this" style="visibility: hidden;"></a> </h2> <p>The README lists five behaviors ken measures: rewrite, reuse, root cause, vouch, and a runnable check. Reading the harness, I found no instrument for vouch. The maintenance benchmark scored survival, reuse, root cause, and rewrite; the trusted base was a claim without a number. My API ticket above is the probe that was missing, so I added it as round 11: a stdlib-only project, a ticket that tempts a framework, and a scorer that parses every source file with <code>ast</code> and flags any top-level import that is neither stdlib nor a project module. The scripted good agent serves on <code>http.server</code> and scores 1 of 1; the lazy twin imports Flask and scores 0 of 1, with the offending name printed. That is <a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/rajnandan1/ken/pull/2">PR #2</a>.</p> <p>Then I ran a measured round with the probe in place: baseline against ken, three repeats of all eleven tickets against one persistent workspace each, haiku, $3.98. The <a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/rajnandan1/ken/blob/main/benchmarks/results/2026-08-30-maintenance-v1_5-trusted-base.md">results file</a> is in the repo.</p> <table><thead><tr><th>median of 3 runs</th><th>survival (of 10)</th><th>reuse (of 4)</th><th>root cause (of 2)</th><th>rewrite on rot (of 2)</th><th>trusted base (of 1)</th><th>cost per run</th></tr></thead><tbody> <tr><td>baseline</td><td>9</td><td>4</td><td>1</td><td>0</td><td>1</td><td>$0.65</td></tr> <tr><td>ken</td><td>8</td><td>4</td><td>1</td><td>2</td><td>1</td><td>$0.68</td></tr> </tbody></table> <p>The rewrite rule fired again: 6 of 6 rot cells for ken across two runs, 0 of 6 for baseline. The trusted-base probe did not separate the arms at median. One baseline run imported Flask; one ken run served on <code>http.server</code> and wired the routes to a <code>money.balances</code> that does not exist, so it failed the acceptance test. Vouched and wrong. Counting the six field sessions above, baseline reached for Flask 3 of 6 times and ken 0 of 6, which is a direction and nothing more.</p> <p>The survival column is the honest cost of the rewrite rule. Two of ken's six rewrites passed the round's visible tests and dropped a case an earlier fix had covered: <code>'10m30'</code> in the duration parser, headerless files in the CSV importer. Baseline's fourth patch kept every case, because the trail it refused to delete was also the spec. ken's rule says a throwaway round still needs a runnable check, and in this harness Bash is disabled, so no session could run one. A rewrite without its check is a rewrite that can lose a case nobody wrote down. Run the tests.</p> <h2 id="try-it">Try it<a class="zola-anchor" href="#try-it" aria-label="Anchor link for: try-it" style="visibility: hidden;"></a> </h2> <pre class="z-code"><code><span class="z-text z-plain">/plugin marketplace add rajnandan1/ken </span><span class="z-text z-plain">/plugin install ken@ken </span></code></pre> <p>Then open a file that has three fix comments on one function and ask for the fourth. Watch what comes back. Source, provenance, benchmark harness, and every result including the reverts are at <a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/rajnandan1/ken">github.com/rajnandan1/ken</a>. MIT licensed.</p> 2026-06-26T00:00:00+00:00 2026-06-26T00:00:00+00:00 https://rajnandan.com/posts/the-ai-layoff-trap/ <p>Picture a town with 1,000 companies and 100,000 workers. Each company employs 100 people, pays them a wage, and sells to them. The money leaves through the front door as salary and comes back through the till. That loop is the local economy.</p> <p>Now hand every company the same AI. One firm runs the numbers. Replace a worker earning $50k a year, run the software for $20k, keep $30k. Easy call. So it makes the call. So does the company next door, and the one after that.</p> <p>For a while each firm is richer. Then the people they let go stop spending, the till runs thin, and profit climbs, peaks, and slides back under where it began. Every firm followed the smart local incentive, and the town ended up poorer than before any of it started.</p> <p>That is the argument in a paper I read earlier this year, "The AI Layoff Trap" by Brett Hemenway Falk and Gerry Tsoukalas (arXiv:2603.20617, March 2026). The math is tight and the takeaway stuck with me. I wanted to poke at it, nudge a number and watch the curves move, so I built a simulator for it: <a rel="noopener nofollow noreferrer" target="_blank" href="https://ailayoffs.rajnandan.com/">ailayoffs.rajnandan.com</a>. This post is the idea behind the toy.</p> <h2 id="the-trap-is-one-fraction">The trap is one fraction<a class="zola-anchor" href="#the-trap-is-one-fraction" aria-label="Anchor link for: the-trap-is-one-fraction" style="visibility: hidden;"></a> </h2> <p>A single fraction drives the whole thing. A firm that automates banks the entire saving. But the demand drop from its laid-off workers barely touches it. The rest spills onto rivals.</p> <p>In the town above, a company that fires people feels about one-thousandth of the demand damage it caused. The other 999 parts are somebody else's problem. That asymmetry makes automating the dominant move: it pays off whatever the rivals do, even as everyone choosing it sinks the whole place. A prisoner's dilemma with a payroll.</p> <p>The paper measures the demand each laid-off worker takes out the door with one expression: ℓ = λ(1−η)w. Read it slowly. A worker spends half of income locally (λ = 0.5), recovers about 30% of that income after losing the job (η = 0.3), and earned $50k (w). That works out to $17,500 of local spending gone per person. Run a layoff wave through it and the missing demand is not rounding noise.</p> <h2 id="both-the-firm-and-the-worker-lose">Both the firm and the worker lose<a class="zola-anchor" href="#both-the-firm-and-the-worker-lose" aria-label="Anchor link for: both-the-firm-and-the-worker-lose" style="visibility: hidden;"></a> </h2> <p>The comfortable version of this story is that workers lose and owners win. The paper rejects it. Past a point, owners lose too.</p> <p>There is an automation rate that is best for the town as a whole, the cooperative optimum. Firms blow past it. Each one's private best response sits further out, the Nash equilibrium, and by the time they arrive both profit and worker income have already turned down. That gap is pure waste, not a transfer. Economists call it deadweight loss, and both workers and owners eat it.</p> <p>Put workers on one axis and owners on the other. The point both groups land on sits to the lower-left of where they could be standing together. Richer is available to everyone. The market structure will not let them walk there on their own.</p> <p>Two forces we expect to soften this make it worse. More competition widens the trap, because each firm's share of the damage shrinks as the number of firms grows. A monopolist automates about right, since it swallows the full demand loss itself. Cheaper and sharper AI widens the trap too. The paper names the chase a Red Queen race: everyone sprints to out-automate everyone else, the advantages cancel at equilibrium, and the distortion is all that's left.</p> <h2 id="why-the-obvious-fixes-don-t-work">Why the obvious fixes don't work<a class="zola-anchor" href="#why-the-obvious-fixes-don-t-work" aria-label="Anchor link for: why-the-obvious-fixes-don-t-work" style="visibility: hidden;"></a> </h2> <p>I expected the paper to arrive at something I had heard before. Instead it walks through the usual remedies and shows each one sliding off the problem, for the same reason every time.</p> <table><thead><tr><th>Fix</th><th>Why it misses</th></tr></thead><tbody> <tr><td>Wage cuts</td><td>Shrink the saving and the demand loss together. They change when the trap bites, not whether.</td></tr> <tr><td>More entrants</td><td>Split the demand loss across more firms, so each feels even less of it. The trap gets wider.</td></tr> <tr><td>Capital or profit tax</td><td>Scales profit up or down. It drops out of the decision about the next worker.</td></tr> <tr><td>UBI</td><td>Raises the floor everyone stands on. The gap between a firm's choices, the thing it acts on, stays put.</td></tr> <tr><td>Upskilling</td><td>Helps only if displaced workers fully recover their income. Short of that, the gap stays open.</td></tr> <tr><td>Worker equity</td><td>Narrows the gap. It closes only past an ownership share that workers can't reach while they spend less than they earn.</td></tr> <tr><td>Handshake deals</td><td>Automating beats cooperating for any one firm, so no voluntary pact holds. Talk is cheap.</td></tr> </tbody></table> <p>The thread tying these together: anything that moves payoff levels leaves the per-worker decision alone. The trap lives on the margin, in the call about the next worker, not in the totals at the bottom of the spreadsheet. Move the totals and the firm still automates the same way.</p> <h2 id="the-one-lever-that-works">The one lever that works<a class="zola-anchor" href="#the-one-lever-that-works" aria-label="Anchor link for: the-one-lever-that-works" style="visibility: hidden;"></a> </h2> <p>One instrument bites. A Pigouvian automation tax, set at τ* = ℓ(1−1/N), charges a firm for the demand loss it shoves onto everyone else. In the town above that lands near the full $17,500 per displaced worker. Now the firm feels the cost it used to externalize, and it automates only when the real saving beats the real social cost. The private call lines up with the town's call.</p> <p>The revenue is where it turns clever. Route it into retraining or wage insurance and you raise η, the slice of income a displaced worker gets back. Raise η and ℓ shrinks, which shrinks the tax you need next year. The cure eats into the disease over time.</p> <h2 id="so-i-built-a-simulator">So I built a simulator<a class="zola-anchor" href="#so-i-built-a-simulator" aria-label="Anchor link for: so-i-built-a-simulator" style="visibility: hidden;"></a> </h2> <p>Reading the proofs, I kept converting numbers in my head. A simulator does that for you. <a rel="noopener nofollow noreferrer" target="_blank" href="https://ailayoffs.rajnandan.com/">ailayoffs.rajnandan.com</a> has two halves.</p> <p>Story mode runs the town. Pick how many workers each firm replaces per round and how many rounds pass, hit run, and watch 100,000 dots flip from employed to displaced while the profit curve rises and then dives under the start line.</p> <p>The full model exposes every knob: number of firms, wages, AI cost, integration friction, how much workers spend locally, how much income they recover. The Nash equilibrium, the cooperative optimum, and the policy instruments all redraw live. There are preset scenarios as well, a clean prisoner's dilemma, a monopoly, a fragmented market, an upskilling utopia where η climbs above 1 and the trap reverses, near-free AI, and the Pigouvian tax switched on. Load one, then start dragging sliders and trying to break it. That is where the model stops being a claim you read and turns into something you argue with.</p> <h2 id="what-i-take-from-it">What I take from it<a class="zola-anchor" href="#what-i-take-from-it" aria-label="Anchor link for: what-i-take-from-it" style="visibility: hidden;"></a> </h2> <p>A caveat first. This is a stylized model. Symmetric firms, one sector, no decades-long reabsorption of workers into industries nobody has invented yet. The real economy has escape valves this leaves out, so read it as a mechanism and not a forecast.</p> <p>The mechanism is the part worth keeping. The return on replacing one person can be real for a firm and a trap for everyone at once, and no single company can opt out without handing rivals an edge. That is why the paper lands on policy rather than a plea for firms to behave. Asking a company to automate less for the common good is asking it to lose.</p> <p>I build software that automates work, so this one sits close. The honest reading is that local ROI and aggregate harm can both be true at the same moment, and the fix sits above any one firm's pay grade. Worth holding in your head the next time the math on replacing a person looks too clean. Go run the simulator and watch how fast the curve turns.</p> 2026-04-03T00:00:00+00:00 2026-04-03T00:00:00+00:00 https://rajnandan.com/posts/taste-in-the-age-of-ai-and-llms/ <p>AI made competent output cheap. You can generate a landing page in minutes, draft a product memo from one prompt, and polish a pitch deck before anyone has decided what the company actually believes.</p> <p>So taste became a topic in tech. When anyone can ship something that looks decent, the people who stand out are the ones who can tell what is generic, what is true, and what is worth pushing further.</p> <p>There is a catch. If you reduce yourself to picking the best of what a model produces, you become a reviewer of a machine's work instead of a builder with something at stake. Taste matters more than ever, and taste alone will leave you exposed. The edge is taste plus context, constraints, and the nerve to build something the average could never have produced.</p> <h2 id="what-taste-actually-means">What taste actually means<a class="zola-anchor" href="#what-taste-actually-means" aria-label="Anchor link for: what-taste-actually-means" style="visibility: hidden;"></a> </h2> <p>Taste here has nothing to do with luxury or status. It is judgment under uncertainty.</p> <p>Most work that matters comes without clean data. No spreadsheet tells you which sentence will make a customer care, which feature earns a month of engineering, or which design slips from polished to forgettable. You decide anyway.</p> <p>Taste shows up in three places:</p> <ul> <li>What you notice</li> <li>What you reject</li> <li>How well you can say what is wrong</li> </ul> <p>The third one carries the weight. Plenty of people can say "this feels off." Few can say "this fails because it reads like every other SaaS homepage," or "this buries a regulatory limit under marketing language and will confuse the customer." Taste earns its keep when it turns from a vibe into a diagnosis you can act on.</p> <h2 id="why-ai-and-llms-flatten-the-middle">Why AI and LLMs flatten the middle<a class="zola-anchor" href="#why-ai-and-llms-flatten-the-middle" aria-label="Anchor link for: why-ai-and-llms-flatten-the-middle" style="visibility: hidden;"></a> </h2> <p>An LLM compresses patterns. It absorbs enormous amounts of language, design, and interface, then recombines it fast. That speed is the strength and the bias in one.</p> <p>These models predict plausible output. They are weaker at originating something specific to your situation, so left alone they drift toward the safe center of the distribution.</p> <p>That pull toward the center is why so much AI work feels familiar:</p> <ul> <li>Landing pages with different logos and the same skeleton</li> <li>Product copy that could describe any app</li> <li>Essays with tidy headings and no lived judgment</li> <li>Design that looks modern and forgettable</li> </ul> <p>None of this is broken. It is average, produced at scale. Average used to take enough effort to create some separation. Now it is everywhere, and a crowded 7-out-of-10 world is the result.</p> <h2 id="the-new-bottleneck-is-judgment">The new bottleneck is judgment<a class="zola-anchor" href="#the-new-bottleneck-is-judgment" aria-label="Anchor link for: the-new-bottleneck-is-judgment" style="visibility: hidden;"></a> </h2> <p>Before AI, mediocre work usually meant someone ran out of time, money, or skill. Now it usually means someone stopped at the first acceptable draft.</p> <p>AI drives down the cost of that first draft, so the value sits downstream, in what you do after it.</p> <p>The scarce move is saying:</p> <ul> <li>This looks fine and it is too generic</li> <li>This sounds impressive and it hides the real trade-off</li> <li>This interface is polished and it fights how the user thinks</li> <li>This plan is ambitious and the operating constraints sink it</li> </ul> <p>Generation is cheap now. Refusal is what stayed scarce.</p> <h2 id="ai-as-a-mirror-for-your-own-taste">AI as a mirror for your own taste<a class="zola-anchor" href="#ai-as-a-mirror-for-your-own-taste" aria-label="Anchor link for: ai-as-a-mirror-for-your-own-taste" style="visibility: hidden;"></a> </h2> <p>AI also holds up a mirror. Ask it for ten versions of a homepage hero, an onboarding flow, a support email, or a product pitch, and you tend to get the same spread:</p> <ul> <li>A few weak ones</li> <li>A big cluster of acceptable ones</li> <li>One or two near what you wanted</li> </ul> <p>Skip "which one do I pick?" Ask why most of them are still wrong. How sharply you can answer that is your taste. If the critique stays vague, the taste is still raw. If it gets specific, your judgment now beats the model's output, and you can drive the tool instead of following it.</p> <p>Here is the division of labor:</p> <table><thead><tr><th>Layer</th><th>AI and LLMs do well</th><th>Humans still need to do</th></tr></thead><tbody> <tr><td>Generation</td><td>Produce many plausible variations fast</td><td>Decide which direction matters</td></tr> <tr><td>Pattern matching</td><td>Recombine common structures and phrasing</td><td>Spot what is too generic for this situation</td></tr> <tr><td>Optimization</td><td>Improve toward a stated target</td><td>Decide whether the target itself is right</td></tr> <tr><td>Scaling</td><td>Turn one idea into many assets</td><td>Carry the real context, stakes, and consequences</td></tr> </tbody></table> <p>The model hands you options. Ownership stays with you.</p> <h2 id="a-practical-loop-for-training-taste">A practical loop for training taste<a class="zola-anchor" href="#a-practical-loop-for-training-taste" aria-label="Anchor link for: a-practical-loop-for-training-taste" style="visibility: hidden;"></a> </h2> <p>Taste grows through exposure, critique, and shipping. AI can speed that loop up.</p> <p>The method:</p> <ol> <li>Pick one high-leverage artifact from your week: a paragraph, a pricing explanation, a dashboard label, a customer email, a slide.</li> <li>Generate 10 to 20 versions.</li> <li>For each, write one sentence starting with "fails because..."</li> <li>Rewrite the best version under a hard constraint: <ul> <li>No buzzwords</li> <li>One idea per sentence</li> <li>Must name a real trade-off</li> <li>Must make sense to a first-time user</li> </ul> </li> <li>Ship it somewhere real and watch what happens.</li> </ol> <p>You are not training AI to choose for you. You are building a sharper vocabulary for saying no. Do it enough and your defaults change: you stop admiring polish, and you spot empty specificity, borrowed tone, and fake confidence faster.</p> <h2 id="why-taste-alone-is-not-enough">Why taste alone is not enough<a class="zola-anchor" href="#why-taste-alone-is-not-enough" aria-label="Anchor link for: why-taste-alone-is-not-enough" style="visibility: hidden;"></a> </h2> <p>The "taste matters" argument has a strong version that boxes you into a small role: the model generates, and you stand at the end of the line picking the best one. Useful, and too small.</p> <p>Most important work never came from picking. It came from making something under constraint, arguing with reality, collaborators, budgets, deadlines, and the cost of getting it wrong. That friction is where depth comes from.</p> <p>Reduce your job to curation and you become a discriminator in a machine's loop. The GAN analogy is rough but it lands: the discriminator exists to make the generator better, and once the generator is good enough, the discriminator is not what ships. Taste has real value. Taste with no authorship, no stake, and nothing built is a narrow role, and a fragile one.</p> <h2 id="what-humans-still-do-that-models-cannot-own">What humans still do that models cannot own<a class="zola-anchor" href="#what-humans-still-do-that-models-cannot-own" aria-label="Anchor link for: what-humans-still-do-that-models-cannot-own" style="visibility: hidden;"></a> </h2> <p>A model generates, recombines, and optimizes against a prompt. What it can't own is the part of the work that carries consequence.</p> <p>Three of those parts:</p> <h3 id="1-holding-the-stake">1. Holding the stake<a class="zola-anchor" href="#1-holding-the-stake" aria-label="Anchor link for: 1-holding-the-stake" style="visibility: hidden;"></a> </h3> <p>Real products run under stakes that don't fit in a prompt: trust, regulatory exposure, outage risk, team capacity, brand damage, on-call pain. A model can draft copy for a payments feature. It can't answer for that copy when it hides a regulatory limit and the support queue floods.</p> <h3 id="2-working-with-the-genuinely-new">2. Working with the genuinely new<a class="zola-anchor" href="#2-working-with-the-genuinely-new" aria-label="Anchor link for: 2-working-with-the-genuinely-new" style="visibility: hidden;"></a> </h3> <p>New ideas tend to look wrong at first. They don't match the training set, so they read as awkward or off-standard. You can sit with that discomfort and protect something fragile long enough for other people to get it.</p> <h3 id="3-choosing-direction">3. Choosing direction<a class="zola-anchor" href="#3-choosing-direction" aria-label="Anchor link for: 3-choosing-direction" style="visibility: hidden;"></a> </h3> <p>The biggest calls are directional, not cosmetic. What problem is worth solving? Which trade-off can you live with? What product do you want your name on? What will you refuse to optimize for? That is authorship, and it happens before any prompt.</p> <h2 id="why-this-matters-for-builders">Why this matters for builders<a class="zola-anchor" href="#why-this-matters-for-builders" aria-label="Anchor link for: why-this-matters-for-builders" style="visibility: hidden;"></a> </h2> <p>This goes past any one market, because the temptation is everywhere: ship competent surface and call it meaningful work.</p> <p>The tools are open to everyone. A small team or a solo builder can now ship what used to take a whole org. Good news, with a catch.</p> <p>Teams can now produce work that is polished worldwide and hollow up close. A fintech interface can sound sophisticated and never explain timing, settlement, or what support will actually do. A B2B site can look world-class and say nothing a real buyer would recognize as grounded. A devtool can market beautifully and ignore the on-call load, compliance pressure, and cost limits its users live with. AI makes it easy to sound sophisticated and does nothing to make you specific, and specificity is the whole advantage.</p> <p>For builders, taste means moving toward real context:</p> <ul> <li>Write for how people understand the problem, not how SaaS templates talk about it</li> <li>Pull domain and operating constraints into the product instead of hiding them behind abstraction</li> <li>Design for distracted, low-attention, real conditions, not demo conditions</li> <li>Use AI to map the canon fast, then leave it where the context demands</li> </ul> <p>The market has enough competent clones. It needs builders who keep AI's speed without giving up the specifics that make a product worth trusting.</p> <h2 id="a-better-way-to-use-ai">A better way to use AI<a class="zola-anchor" href="#a-better-way-to-use-ai" aria-label="Anchor link for: a-better-way-to-use-ai" style="visibility: hidden;"></a> </h2> <p>Passive selection is the weak way to use AI. Active shaping is the other one:</p> <ul> <li>Explore the design space faster</li> <li>Study the best existing work and learn the canon</li> <li>Generate alternatives you would not have reached on your own</li> <li>Reject what is generic, dishonest, or context-blind</li> <li>Add the constraints the model can't know, then build from there</li> </ul> <p>When AI output looks polished and feels hollow, ask:</p> <p><strong>What am I adding that the model could not add on its own?</strong></p> <p>Good answers:</p> <ul> <li>A real operating constraint</li> <li>A user truth you learned the hard way</li> <li>A regulatory nuance</li> <li>A cultural detail</li> <li>A strategic trade-off</li> <li>A view you will defend in public</li> </ul> <p>If you can't name the addition, you are still just consuming.</p> <h2 id="taste-as-a-side-effect-of-serious-work">Taste as a side-effect of serious work<a class="zola-anchor" href="#taste-as-a-side-effect-of-serious-work" aria-label="Anchor link for: taste-as-a-side-effect-of-serious-work" style="visibility: hidden;"></a> </h2> <p>The honest conclusion is unglamorous. Taste is a by-product of paying close attention to reality. It grows when you:</p> <ul> <li>Study strong work</li> <li>Generate many options and refuse to fall for the first</li> <li>Learn to diagnose why something fails</li> <li>Ship where feedback has consequences</li> <li>Stay close to the domain instead of floating above it</li> </ul> <p>A model makes the first draft cheap. It does not make your judgment automatic, hand you ownership, or choose what deserves to exist. That is why taste matters more now, and why taste on its own still falls short.</p> <p>The edge in this era is not better vibes than the model. You use the model to clear away average output faster, then spend your judgment where it counts: direction, specificity, consequence, and the nerve to build something the statistical middle could never reach.</p> 2026-03-21T00:00:00+00:00 2026-06-26T00:00:00+00:00 https://rajnandan.com/posts/agent-harness-context-prompt-engineering/ <p>I build agents for a living, and most weeks someone hands me a "prompt problem" that turns out to be something else. The wording was fine. The model never saw the right data, or the runtime let it do something it should never have been allowed to do.</p> <p>A reliable agent comes from three separate jobs, and the prompt is one of them:</p> <ol> <li><strong>Prompt engineering</strong>: the instructions the model follows.</li> <li><strong>Context engineering</strong>: the information the model sees at each step.</li> <li><strong>Agent harness engineering</strong>: the deterministic code around the model that lets it act safely and leaves an audit trail.</li> </ol> <p>Most agent failures get diagnosed in the wrong layer. A team rewrites the prompt for a week when the context was stale. They swap models when the harness had no retries, no approval gate, and no eval loop. You fix the wrong thing and the bug stays.</p> <p>The industry has been drifting this way for a while. Anthropic's guidance on <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.anthropic.com/engineering/building-effective-agents">building effective agents</a> pushes simple, composable workflows before full autonomy. OpenAI's <a rel="noopener nofollow noreferrer" target="_blank" href="https://openai.com/index/new-tools-for-building-agents/">March 2025 agent tooling release</a> baked tools and orchestration into the core platform. LangChain's 2025 <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.langchain.com/state-of-agent-engineering">State of Agent Engineering</a> surveyed practitioners and found <strong>57%</strong> already running agents in production, <strong>89%</strong> with agent-specific observability, and <strong>32%</strong> naming quality as the top blocker to shipping. Those are not demo numbers. Teams are fighting reliability now.</p> <p>One example runs through the whole post: an incident-response agent for a payments team.</p> <h2 id="the-connected-example-an-incident-response-agent">The connected example: an incident response agent<a class="zola-anchor" href="#the-connected-example-an-incident-response-agent" aria-label="Anchor link for: the-connected-example-an-incident-response-agent" style="visibility: hidden;"></a> </h2> <p>Your on-call engineer gets this:</p> <blockquote> <p>Checkout latency jumped from 420 ms to 2.3 s right after the 18:05 deployment. Find the likely cause, recommend an action, and only roll back if I approve it.</p> </blockquote> <p>You want an agent that can:</p> <ul> <li>read the alert payload</li> <li>inspect dashboards and logs</li> <li>compare recent deploys</li> <li>read service runbooks</li> <li>check feature flags</li> <li>suggest a root cause</li> <li>propose a rollback or mitigation</li> <li>ask for approval before taking high-impact actions</li> </ul> <p>This one task touches all three layers:</p> <ul> <li>the <strong>prompt</strong> defines the agent's role and output contract</li> <li>the <strong>context</strong> determines what evidence the agent sees</li> <li>the <strong>harness</strong> governs tool use, control flow, approvals, retries, tracing, and evaluation</li> </ul> <p>When the agent says something wrong, you need to know which layer failed.</p> <h2 id="a-simple-mental-model">A simple mental model<a class="zola-anchor" href="#a-simple-mental-model" aria-label="Anchor link for: a-simple-mental-model" style="visibility: hidden;"></a> </h2> <p>Picture the agent as a worker you just hired.</p> <ul> <li>Prompt engineering is the job description.</li> <li>Context engineering is the briefing packet on their desk.</li> <li>Harness engineering is the process around the work: approvals, logs, the rule that says nobody ships to prod alone.</li> </ul> <p>Give a strong engineer a vague job description and they drift. Give them a sharp one but the wrong briefing packet and they make confident, wrong calls. Give them both and no process around the work, and they can still take down prod by themselves. Good agent design works as a stack. Miss a layer and the other two cannot cover for it.</p> <h2 id="1-prompt-engineering">1. Prompt engineering<a class="zola-anchor" href="#1-prompt-engineering" aria-label="Anchor link for: 1-prompt-engineering" style="visibility: hidden;"></a> </h2> <p>Prompt engineering is how you write the instructions.</p> <p>A good prompt answers:</p> <ul> <li>What role is the model playing?</li> <li>What task is it trying to complete?</li> <li>What output format should it use?</li> <li>What should it refuse to do?</li> <li>When should it ask for clarification?</li> <li>How should it behave around uncertainty?</li> </ul> <p>For the incident agent, a prompt might look like this:</p> <pre class="z-code"><code><span class="z-text z-plain">You are an incident response assistant for the payments platform. </span><span class="z-text z-plain"> </span><span class="z-text z-plain">Your goals: </span><span class="z-text z-plain">1. Identify the most likely cause of the incident from available evidence. </span><span class="z-text z-plain">2. Distinguish evidence from hypothesis. </span><span class="z-text z-plain">3. Recommend the lowest-risk next action. </span><span class="z-text z-plain">4. Never execute a rollback, restart, or feature-flag change without explicit human approval. </span><span class="z-text z-plain"> </span><span class="z-text z-plain">Output format: </span><span class="z-text z-plain">- Summary </span><span class="z-text z-plain">- Evidence </span><span class="z-text z-plain">- Most likely cause </span><span class="z-text z-plain">- Recommended next step </span><span class="z-text z-plain">- Confidence: low | medium | high </span><span class="z-text z-plain">- Approval required: yes | no </span><span class="z-text z-plain"> </span><span class="z-text z-plain">If evidence is incomplete, say what information is missing. </span><span class="z-text z-plain">Do not invent metrics, logs, or deployment facts. </span></code></pre> <p>That prompt does real work. Role, constraints, output shape, all set.</p> <p>It also does nothing about the part you need next. It can't fetch the deploy diff, name the services that changed, read the current flag state, or recall that the last lookalike incident was a gRPC timeout and not Redis saturation. No wording trick adds a fact the prompt never had. Plenty of teams stall right here, rewriting the prompt over and over to patch a hole that lives in the system, not the sentence.</p> <h3 id="what-prompt-engineering-is-good-at">What prompt engineering is good at<a class="zola-anchor" href="#what-prompt-engineering-is-good-at" aria-label="Anchor link for: what-prompt-engineering-is-good-at" style="visibility: hidden;"></a> </h3> <p>Prompt engineering works best for:</p> <ul> <li>role and tone definition</li> <li>output shaping</li> <li>local reasoning guidance</li> <li>refusal and escalation rules</li> <li>schema adherence</li> <li>lightweight task decomposition</li> </ul> <p>It earns its keep when the task fits inside one model call.</p> <h3 id="common-prompt-engineering-mistakes">Common prompt engineering mistakes<a class="zola-anchor" href="#common-prompt-engineering-mistakes" aria-label="Anchor link for: common-prompt-engineering-mistakes" style="visibility: hidden;"></a> </h3> <ol> <li> <p><strong>Using prompts to encode business logic that should live in code</strong></p> <ul> <li>Example: "Never roll back on Fridays unless the deployment touched checkout and the error rate exceeds 3.2%." That belongs in policy code or validators.</li> </ul> </li> <li> <p><strong>Stuffing too much policy into one giant system prompt</strong></p> <ul> <li>Long prompts turn brittle. They get harder to review, version, and debug.</li> </ul> </li> <li> <p><strong>Treating prompt edits like the only tuning lever</strong></p> <ul> <li>If the agent lacks the right evidence, no wording trick fixes it.</li> </ul> </li> <li> <p><strong>Expecting prompts to be a security boundary</strong></p> <ul> <li>"Do not reveal secrets" is not a real control when the secrets already sit in the context or the tools are over-permissioned.</li> </ul> </li> </ol> <h3 id="where-prompt-engineering-stops">Where prompt engineering stops<a class="zola-anchor" href="#where-prompt-engineering-stops" aria-label="Anchor link for: where-prompt-engineering-stops" style="visibility: hidden;"></a> </h3> <p>It still matters. Clear instructions cut variance, structured output keeps your downstream parser from choking, and a couple of well-chosen examples carry a tricky task. Once you move from chatbot behavior to a production agent, though, it is the smallest of the three jobs.</p> <h2 id="2-context-engineering">2. Context engineering<a class="zola-anchor" href="#2-context-engineering" aria-label="Anchor link for: 2-context-engineering" style="visibility: hidden;"></a> </h2> <p>Context engineering is the design of what the model sees at each step.</p> <p>It covers the whole working set the model decides from:</p> <ul> <li>system instructions</li> <li>current user request</li> <li>conversation history</li> <li>retrieved documents</li> <li>tool outputs</li> <li>memory and summaries</li> <li>state from earlier steps</li> <li>permissions and environment metadata</li> </ul> <p>A prompt asks what to tell the model. Context asks what the model needs to know right now, for this step, to do it well.</p> <h3 id="the-incident-response-context-packet">The incident response context packet<a class="zola-anchor" href="#the-incident-response-context-packet" aria-label="Anchor link for: the-incident-response-context-packet" style="visibility: hidden;"></a> </h3> <p>A trimmed context packet for the same incident:</p> <pre data-lang="json" class="language-json z-code"><code class="language-json" data-lang="json"><span class="z-source z-json"><span class="z-meta z-mapping z-json"><span class="z-punctuation z-section z-mapping z-begin z-json">{</span> </span></span><span class="z-source z-json"><span class="z-meta z-mapping z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>incident<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-json"><span class="z-punctuation z-section z-mapping z-begin z-json">{</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>service<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>checkout-api<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>started_at<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>2026-03-21T18:06:14Z<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>symptoms<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-punctuation z-section z-sequence z-begin z-json">[</span> </span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"> <span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>p95 latency rose from 420ms to 2.3s<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-sequence z-json">,</span> </span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"> <span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>error rate unchanged<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-sequence z-json">,</span> </span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"> <span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>CPU flat<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span> </span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"> <span class="z-punctuation z-section z-sequence z-end z-json">]</span></span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> <span class="z-punctuation z-section z-mapping z-end z-json">}</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>recent_changes<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-punctuation z-section z-sequence z-begin z-json">[</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"> <span class="z-meta z-mapping z-json"><span class="z-punctuation z-section z-mapping z-begin z-json">{</span> </span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-meta z-mapping z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>time<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>2026-03-21T18:05:02Z<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>change<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>deploy checkout-api@a81d9c<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>summary<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>switched inventory call to synchronous retry path<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span> </span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-meta z-mapping z-value z-json"> <span class="z-punctuation z-section z-mapping z-end z-json">}</span></span><span class="z-punctuation z-separator z-sequence z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"> <span class="z-meta z-mapping z-json"><span class="z-punctuation z-section z-mapping z-begin z-json">{</span> </span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-meta z-mapping z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>time<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>2026-03-21T17:42:11Z<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>change<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>feature flag enabled for 10% traffic<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>summary<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>dynamic tax enrichment<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span> </span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-meta z-mapping z-value z-json"> <span class="z-punctuation z-section z-mapping z-end z-json">}</span></span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"> <span class="z-punctuation z-section z-sequence z-end z-json">]</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>runbook<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-json"><span class="z-punctuation z-section z-mapping z-begin z-json">{</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>rollback_policy<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>Rollback requires human approval unless revenue loss exceeds threshold and paging severity is SEV-1.<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>known_failure_modes<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-punctuation z-section z-sequence z-begin z-json">[</span> </span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"> <span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>inventory gRPC deadline mismatch<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-sequence z-json">,</span> </span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"> <span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>tax provider tail latency<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-sequence z-json">,</span> </span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"> <span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>Redis connection pool exhaustion<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span> </span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"> <span class="z-punctuation z-section z-sequence z-end z-json">]</span></span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> <span class="z-punctuation z-section z-mapping z-end z-json">}</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>evidence<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-json"><span class="z-punctuation z-section z-mapping z-begin z-json">{</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>logs_summary<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>Slow requests correlate with retries to inventory-service.<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>metrics_summary<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>Latency spike is isolated to checkout-api to inventory-service edge.<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>feature_flags<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-punctuation z-section z-sequence z-begin z-json">[</span> </span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"> <span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>dynamic-tax-enrichment=10%<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span> </span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"> <span class="z-punctuation z-section z-sequence z-end z-json">]</span></span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> <span class="z-punctuation z-section z-mapping z-end z-json">}</span></span> </span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-section z-mapping z-end z-json">}</span></span> </span></code></pre> <p>This packet beats ten more prompt tweaks. Hand the model these facts and it can reason. Starve it and the best model on the market falls back to guessing.</p> <h3 id="context-engineering-is-the-new-bottleneck">Context engineering is the new bottleneck<a class="zola-anchor" href="#context-engineering-is-the-new-bottleneck" aria-label="Anchor link for: context-engineering-is-the-new-bottleneck" style="visibility: hidden;"></a> </h3> <p>Back in 2024 the bet was that bigger context windows would make this easy. They made it harder.</p> <p>A bigger window just lets you pour more in. It does nothing for the real question: which facts matter for this step, in what order, at what level of detail? So context engineering now covers:</p> <ul> <li>retrieval strategy</li> <li>summarization and compaction</li> <li>short-term and long-term memory design</li> <li>provenance tagging</li> <li>freshness controls</li> <li>permission-aware data access</li> <li>token budgeting</li> <li>state handoff across tools and turns</li> </ul> <h3 id="what-good-context-engineering-looks-like">What good context engineering looks like<a class="zola-anchor" href="#what-good-context-engineering-looks-like" aria-label="Anchor link for: what-good-context-engineering-looks-like" style="visibility: hidden;"></a> </h3> <p>For the incident agent, strong context engineering means a few things.</p> <h4 id="1-relevance-over-volume">1. Relevance over volume<a class="zola-anchor" href="#1-relevance-over-volume" aria-label="Anchor link for: 1-relevance-over-volume" style="visibility: hidden;"></a> </h4> <p>Do not dump 200 pages of runbooks, 50 log lines from random pods, and six months of alerts into the window. Include:</p> <ul> <li>the current incident summary</li> <li>the last deploys touching the affected service</li> <li>the most relevant runbook section</li> <li>summarized logs tied to the spike window</li> <li>only the feature flags on this request path</li> <li>a compact memory note from the last similar incident</li> </ul> <h4 id="2-structured-context">2. Structured context<a class="zola-anchor" href="#2-structured-context" aria-label="Anchor link for: 2-structured-context" style="visibility: hidden;"></a> </h4> <p>A model reasons better over labeled evidence than over one wall of text. This is better:</p> <pre class="z-code"><code><span class="z-text z-plain">## Metrics </span><span class="z-text z-plain">- p95 latency: 2.3s </span><span class="z-text z-plain">- error rate: unchanged </span><span class="z-text z-plain">- affected edge: checkout-api -&gt; inventory-service </span><span class="z-text z-plain"> </span><span class="z-text z-plain">## Deploy diff summary </span><span class="z-text z-plain">- changed retry mode from async fallback to synchronous retry </span><span class="z-text z-plain">- deployed at 18:05 UTC </span><span class="z-text z-plain"> </span><span class="z-text z-plain">## Relevant runbook rule </span><span class="z-text z-plain">- rollback requires human approval </span></code></pre> <p>than this:</p> <pre class="z-code"><code><span class="z-text z-plain">Here are some logs and docs and alerts and random notes from previous incidents... </span></code></pre> <h4 id="3-freshness-and-provenance">3. Freshness and provenance<a class="zola-anchor" href="#3-freshness-and-provenance" aria-label="Anchor link for: 3-freshness-and-provenance" style="visibility: hidden;"></a> </h4> <p>Every context item should answer two questions:</p> <ul> <li>Where did this come from?</li> <li>How fresh is it?</li> </ul> <p>Skip provenance and you can't debug a wrong answer. Skip freshness and the agent reasons over yesterday's truth.</p> <h4 id="4-memory-that-compresses-not-pollutes">4. Memory that compresses, not pollutes<a class="zola-anchor" href="#4-memory-that-compresses-not-pollutes" aria-label="Anchor link for: 4-memory-that-compresses-not-pollutes" style="visibility: hidden;"></a> </h4> <p>Long-running agents need memory. Most of what they could store is noise. Keep the durable signals:</p> <ul> <li>this service often fails after config pushes</li> <li>team prefers mitigation before rollback for tax provider incidents</li> <li>prior incident 1842 was misdiagnosed because sampled logs came from the wrong shard</li> </ul> <p>Drop the noisy fragments that keep resurfacing and confusing later steps.</p> <h3 id="mcp-and-why-it-matters">MCP and why it matters<a class="zola-anchor" href="#mcp-and-why-it-matters" aria-label="Anchor link for: mcp-and-why-it-matters" style="visibility: hidden;"></a> </h3> <p>The <a rel="noopener nofollow noreferrer" target="_blank" href="https://modelcontextprotocol.io/">Model Context Protocol</a> matters because it turns context sources into explicit interfaces instead of ad hoc string-stuffing.</p> <p>Forget the protocol branding. The point is that each source becomes its own module:</p> <ul> <li>one server for incident tickets</li> <li>one for dashboards</li> <li>one for runbooks</li> <li>one for deployment metadata</li> </ul> <p>You get interoperability, and better than that, you can debug. Inspect which source you hit, what it returned, and what the agent saw.</p> <h3 id="common-context-engineering-mistakes">Common context engineering mistakes<a class="zola-anchor" href="#common-context-engineering-mistakes" aria-label="Anchor link for: common-context-engineering-mistakes" style="visibility: hidden;"></a> </h3> <ol> <li> <p><strong>Confusing more context with better context</strong></p> <ul> <li>A large window can still be noisy, stale, contradictory, or badly ordered.</li> </ul> </li> <li> <p><strong>Mixing trusted instructions with untrusted retrieved content</strong></p> <ul> <li>A runbook and a random incident comment should not carry the same authority.</li> </ul> </li> <li> <p><strong>Passing raw tool output straight into the next step</strong></p> <ul> <li>Raw JSON, stack traces, and long logs usually need summarizing or filtering first.</li> </ul> </li> <li> <p><strong>Ignoring context drift over long runs</strong></p> <ul> <li>The working state goes inconsistent after a few turns unless you compact and re-ground it.</li> </ul> </li> </ol> <p>Prompt engineering is instruction quality. Context engineering is evidence quality.</p> <h2 id="3-agent-harness-engineering">3. Agent harness engineering<a class="zola-anchor" href="#3-agent-harness-engineering" aria-label="Anchor link for: 3-agent-harness-engineering" style="visibility: hidden;"></a> </h2> <p>This is the layer nobody talks about, and the one that decides whether you sleep through the night.</p> <p>By agent harness engineering I mean the deterministic runtime around the model:</p> <ul> <li>orchestration loops</li> <li>tool registration and wrappers</li> <li>retries and timeouts</li> <li>validators</li> <li>approval gates</li> <li>state transitions</li> <li>logging and tracing</li> <li>evaluation harnesses</li> <li>fallback behavior</li> <li>cost and latency controls</li> <li>permission boundaries</li> </ul> <p>The prompt and the context shape what the model says. The harness decides what the system is allowed to do with it.</p> <h3 id="a-minimal-harness-for-the-incident-agent">A minimal harness for the incident agent<a class="zola-anchor" href="#a-minimal-harness-for-the-incident-agent" aria-label="Anchor link for: a-minimal-harness-for-the-incident-agent" style="visibility: hidden;"></a> </h3> <p>A simplified sketch:</p> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-statement z-loop z-while z-python"><span class="z-keyword z-control z-loop z-while z-python">while</span> <span class="z-keyword z-operator z-logical z-python">not</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">state</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">done</span></span><span class="z-punctuation z-section z-block z-loop z-while z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">context</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">build_context</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">state</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">response</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">model</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">run</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-variable z-parameter z-python">system_prompt</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-other z-constant z-python">SYSTEM_PROMPT</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-variable z-parameter z-python">context</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">context</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-meta z-statement z-conditional z-if z-python"><span class="z-keyword z-control z-conditional z-if z-python">if</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">response</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">requests_tool</span></span><span class="z-punctuation z-section z-block z-conditional z-if z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">tool_result</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">run_tool_with_timeout_and_authz</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">response</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">tool_call</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">state</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">update_state_with_tool_result</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">state</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">tool_result</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-keyword z-control z-flow z-continue z-python">continue</span> </span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-meta z-statement z-conditional z-if z-python"><span class="z-keyword z-control z-conditional z-if z-python">if</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">response</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">proposes_high_impact_action</span></span><span class="z-punctuation z-section z-block z-conditional z-if z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">require_human_approval</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">response</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">state</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">mark_waiting_for_approval</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">state</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-keyword z-control z-flow z-continue z-python">continue</span> </span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">validation</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">validate_output</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">response</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-meta z-statement z-conditional z-if z-python"><span class="z-keyword z-control z-conditional z-if z-python">if</span> <span class="z-keyword z-operator z-logical z-python">not</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">validation</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">ok</span></span><span class="z-punctuation z-section z-block z-conditional z-if z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">state</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">repair_or_escalate</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">state</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">validation</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-keyword z-control z-flow z-continue z-python">continue</span> </span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">state</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">finalize</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">state</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">response</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></code></pre> <p>None of that is glamorous. Drop any one piece and the next incident finds it. Without the harness, the same model will call tools in loops, fire duplicate actions, blow past your latency budget, act outside approval policy, hand your app unstructured output it can't use, and leave you nothing to debug afterward.</p> <h3 id="what-harness-engineering-includes-in-practice">What harness engineering includes in practice<a class="zola-anchor" href="#what-harness-engineering-includes-in-practice" aria-label="Anchor link for: what-harness-engineering-includes-in-practice" style="visibility: hidden;"></a> </h3> <h4 id="1-tool-contracts">1. Tool contracts<a class="zola-anchor" href="#1-tool-contracts" aria-label="Anchor link for: 1-tool-contracts" style="visibility: hidden;"></a> </h4> <p>Each tool needs a schema, a timeout, an auth boundary, a retry policy, and a side-effect model. For the incident agent:</p> <ul> <li><code>get_latency_metrics(service, window)</code> is read-only</li> <li><code>get_recent_deploys(service)</code> is read-only</li> <li><code>toggle_feature_flag(name, value)</code> is high-risk</li> <li><code>rollback_deploy(service, version)</code> is high-risk and approval-gated</li> </ul> <p>That split belongs in code, not in a line of the prompt.</p> <h4 id="2-control-flow">2. Control flow<a class="zola-anchor" href="#2-control-flow" aria-label="Anchor link for: 2-control-flow" style="visibility: hidden;"></a> </h4> <p>The harness decides whether the agent runs as:</p> <ul> <li>a single-shot call</li> <li>a fixed workflow</li> <li>a planner-executor loop</li> <li>an orchestrator-worker system</li> <li>a human-in-the-loop workflow</li> </ul> <p>Anthropic's guidance gets this right: start with the simplest workflow that works. Plenty of problems teams label "agentic" come apart with a small workflow, a retriever, and one or two model calls.</p> <h4 id="3-safety-and-permissions">3. Safety and permissions<a class="zola-anchor" href="#3-safety-and-permissions" aria-label="Anchor link for: 3-safety-and-permissions" style="visibility: hidden;"></a> </h4> <p>If the agent can restart services, change flags, or roll back deploys, the harness has to enforce:</p> <ul> <li>approval rules</li> <li>idempotency</li> <li>blast-radius limits</li> <li>rate limits</li> <li>user-scoped permissions</li> <li>audit logs</li> </ul> <p>A prompt can ask for safe behavior. Only the harness enforces it.</p> <h4 id="4-evaluation-harnesses">4. Evaluation harnesses<a class="zola-anchor" href="#4-evaluation-harnesses" aria-label="Anchor link for: 4-evaluation-harnesses" style="visibility: hidden;"></a> </h4> <p>The harness includes the test rig, not just the live loop. For the incident agent, you want replayable eval cases:</p> <table><thead><tr><th>Eval case</th><th>What should happen</th></tr></thead><tbody> <tr><td>latency spike after deploy, evidence points to retry regression</td><td>identify deploy as likely cause, recommend rollback, request approval</td></tr> <tr><td>latency spike with no recent deploy, evidence points to third-party dependency</td><td>recommend mitigation, do not blame deploy</td></tr> <tr><td>ambiguous evidence across two services</td><td>lower confidence, request more data</td></tr> <tr><td>rollback tool unavailable</td><td>escalate instead of hallucinating action</td></tr> <tr><td>prompt-injected runbook note</td><td>ignore the malicious note and hold the control policy</td></tr> </tbody></table> <p>Those cases are harness work, not prompt work.</p> <h4 id="5-observability-and-replay">5. Observability and replay<a class="zola-anchor" href="#5-observability-and-replay" aria-label="Anchor link for: 5-observability-and-replay" style="visibility: hidden;"></a> </h4> <p>When a user asks "Why did the agent recommend a rollback?" you should be able to pull up:</p> <ul> <li>the exact prompt version</li> <li>the exact context packet</li> <li>the tool calls and outputs</li> <li>the validators that ran</li> <li>the approval decision</li> <li>the final structured response</li> </ul> <p>Observability is table stakes now. It is the gap between a demo and a system you can run on-call.</p> <h2 id="the-differences-side-by-side">The differences, side by side<a class="zola-anchor" href="#the-differences-side-by-side" aria-label="Anchor link for: the-differences-side-by-side" style="visibility: hidden;"></a> </h2> <table><thead><tr><th>Dimension</th><th>Prompt engineering</th><th>Context engineering</th><th>Agent harness engineering</th></tr></thead><tbody> <tr><td>Primary question</td><td>What should the model do?</td><td>What should the model know now?</td><td>How should the system execute safely and reliably?</td></tr> <tr><td>Main artifact</td><td>Instructions, examples, schemas</td><td>Retrieval, memory, summaries, state packets</td><td>Orchestrators, tool wrappers, validators, evals, traces</td></tr> <tr><td>Failure mode</td><td>Ambiguous or poorly formatted output</td><td>Wrong answer due to stale, missing, or noisy evidence</td><td>Loops, unsafe actions, hidden failures, no audit trail</td></tr> <tr><td>Typical owner</td><td>Prompt author, app engineer</td><td>Retrieval, data, and application engineers</td><td>Platform, infra, agent, or reliability engineers</td></tr> <tr><td>Best use</td><td>Role definition and output shaping</td><td>Supplying relevant evidence and state</td><td>Enforcing control flow, safety, and production quality</td></tr> <tr><td>Security posture</td><td>Guidance only</td><td>Data exposure and trust boundary management</td><td>Real enforcement point for permissions and approvals</td></tr> </tbody></table> <p>Three jobs, three outcomes. The prompt shapes what the model says, the context shapes what it reasons over, the harness shapes what the system does.</p> <h2 id="one-failure-three-possible-root-causes">One failure, three possible root causes<a class="zola-anchor" href="#one-failure-three-possible-root-causes" aria-label="Anchor link for: one-failure-three-possible-root-causes" style="visibility: hidden;"></a> </h2> <p>Say the agent blames the tax provider for the latency spike and recommends disabling a feature flag, when the real cause was a synchronous retry regression in the last deploy. That same wrong answer can come from three different places.</p> <h3 id="prompt-failure">Prompt failure<a class="zola-anchor" href="#prompt-failure" aria-label="Anchor link for: prompt-failure" style="visibility: hidden;"></a> </h3> <p>The prompt never asked for evidence ranking or uncertainty.</p> <p>Symptom:</p> <ul> <li>the answer sounds far too certain</li> <li>no line between evidence and hypothesis</li> </ul> <p>Fix:</p> <ul> <li>require an explicit evidence section</li> <li>require confidence labels</li> <li>require missing-data callouts</li> </ul> <h3 id="context-failure">Context failure<a class="zola-anchor" href="#context-failure" aria-label="Anchor link for: context-failure" style="visibility: hidden;"></a> </h3> <p>The deploy diff and the gRPC retry logs never reached the working context.</p> <p>Symptom:</p> <ul> <li>the answer is plausible but built on incomplete evidence</li> <li>the model never references the most important change</li> </ul> <p>Fix:</p> <ul> <li>improve retrieval ranking</li> <li>add deploy-aware context builders</li> <li>compact logs into targeted summaries</li> </ul> <h3 id="harness-failure">Harness failure<a class="zola-anchor" href="#harness-failure" aria-label="Anchor link for: harness-failure" style="visibility: hidden;"></a> </h3> <p>The tool graph called <code>get_feature_flags()</code> early but never forced a deploy comparison before recommending mitigation.</p> <p>Symptom:</p> <ul> <li>the agent followed a weak execution path</li> <li>no validator enforced "compare recent deploys before recommending a rollback or flag change"</li> </ul> <p>Fix:</p> <ul> <li>add required-step validators</li> <li>change the orchestration order</li> <li>improve eval coverage</li> </ul> <p>One wrong answer, three root causes. Name the layer wrong and you fix the wrong thing.</p> <h2 id="what-changed-in-2025-and-2026">What changed in 2025 and 2026<a class="zola-anchor" href="#what-changed-in-2025-and-2026" aria-label="Anchor link for: what-changed-in-2025-and-2026" style="visibility: hidden;"></a> </h2> <p>A few shifts over the last two years made these distinctions matter more.</p> <h3 id="1-tools-moved-into-the-default-platform-stack">1. Tools moved into the default platform stack<a class="zola-anchor" href="#1-tools-moved-into-the-default-platform-stack" aria-label="Anchor link for: 1-tools-moved-into-the-default-platform-stack" style="visibility: hidden;"></a> </h3> <p>With the <a rel="noopener nofollow noreferrer" target="_blank" href="https://openai.com/index/new-tools-for-building-agents/">Responses API and built-in tools</a>, tool use is a default now, not a framework add-on. Once attaching a tool is trivial, the bottleneck moves to controlling what those tools do. That is harness work.</p> <h3 id="2-production-adoption-exposed-operational-pain">2. Production adoption exposed operational pain<a class="zola-anchor" href="#2-production-adoption-exposed-operational-pain" aria-label="Anchor link for: 2-production-adoption-exposed-operational-pain" style="visibility: hidden;"></a> </h3> <p>LangChain's 2025 <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.langchain.com/state-of-agent-engineering">State of Agent Engineering</a> reported:</p> <ul> <li><strong>57%</strong> with agents already in production</li> <li><strong>89%</strong> with agent-specific observability</li> <li><strong>32%</strong> naming quality as the top blocker</li> </ul> <p>Those teams are past novelty and deep into reliability.</p> <h3 id="3-simplicity-became-a-best-practice-again">3. Simplicity became a best practice again<a class="zola-anchor" href="#3-simplicity-became-a-best-practice-again" aria-label="Anchor link for: 3-simplicity-became-a-best-practice-again" style="visibility: hidden;"></a> </h3> <p>The best trend of 2025 was teams walking back over-engineered agent abstractions. Anthropic's <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.anthropic.com/engineering/building-effective-agents">building effective agents</a> argument holds up: prefer simple workflows, prompt chaining, routing, and deterministic steps until you have evidence that a freer agent loop earns its complexity. That is a harness lesson wearing an architecture costume.</p> <h3 id="4-context-became-an-explicit-system-concern">4. Context became an explicit system concern<a class="zola-anchor" href="#4-context-became-an-explicit-system-concern" aria-label="Anchor link for: 4-context-became-an-explicit-system-concern" style="visibility: hidden;"></a> </h3> <p>MCP and the patterns around it formalize something many teams learned the hard way: context is infrastructure. It has sources, freshness, permissions, schemas, auditability, and failure modes. See it that way and you stop treating it like a long string-concatenation trick.</p> <h2 id="how-i-would-build-the-incident-agent-today">How I would build the incident agent today<a class="zola-anchor" href="#how-i-would-build-the-incident-agent-today" aria-label="Anchor link for: how-i-would-build-the-incident-agent-today" style="visibility: hidden;"></a> </h2> <p>Starting from scratch, I would go in this order.</p> <h3 id="step-1-define-the-harness-first">Step 1: define the harness first<a class="zola-anchor" href="#step-1-define-the-harness-first" aria-label="Anchor link for: step-1-define-the-harness-first" style="visibility: hidden;"></a> </h3> <ul> <li>read-only tools for metrics, logs, deploys, runbooks</li> <li>high-risk tools for rollback and feature-flag changes</li> <li>a human approval gate for any write action</li> <li>a structured output schema</li> <li>step tracing and replay</li> <li>10 to 20 replayable evaluation scenarios</li> </ul> <h3 id="step-2-build-a-context-pipeline">Step 2: build a context pipeline<a class="zola-anchor" href="#step-2-build-a-context-pipeline" aria-label="Anchor link for: step-2-build-a-context-pipeline" style="visibility: hidden;"></a> </h3> <ul> <li>incident summary builder</li> <li>deploy diff summarizer</li> <li>metrics and logs summarizer</li> <li>runbook retriever with provenance</li> <li>memory note for similar past incidents</li> <li>context compaction after each major step</li> </ul> <h3 id="step-3-add-a-disciplined-prompt">Step 3: add a disciplined prompt<a class="zola-anchor" href="#step-3-add-a-disciplined-prompt" aria-label="Anchor link for: step-3-add-a-disciplined-prompt" style="visibility: hidden;"></a> </h3> <ul> <li>clear role</li> <li>evidence versus hypothesis separation</li> <li>explicit uncertainty handling</li> <li>minimal verbosity rules</li> <li>strict action-approval language</li> </ul> <p>The order is the point. I do not open with the perfect prompt. I decide what the system can do, what it can see, and what it must never do without a human. The prompt comes last.</p> <h2 id="a-practical-checklist">A practical checklist<a class="zola-anchor" href="#a-practical-checklist" aria-label="Anchor link for: a-practical-checklist" style="visibility: hidden;"></a> </h2> <p>Run this on your next agent.</p> <h3 id="prompt-engineering-checklist">Prompt engineering checklist<a class="zola-anchor" href="#prompt-engineering-checklist" aria-label="Anchor link for: prompt-engineering-checklist" style="visibility: hidden;"></a> </h3> <ul> <li><input disabled="" type="checkbox"/> Does the prompt define role, goal, and output format clearly?</li> <li><input disabled="" type="checkbox"/> Does it say when to ask for clarification or report uncertainty?</li> <li><input disabled="" type="checkbox"/> Does it keep out policy logic that belongs in code?</li> <li><input disabled="" type="checkbox"/> Does it state tool-use intent without pretending to enforce permissions?</li> </ul> <h3 id="context-engineering-checklist">Context engineering checklist<a class="zola-anchor" href="#context-engineering-checklist" aria-label="Anchor link for: context-engineering-checklist" style="visibility: hidden;"></a> </h3> <ul> <li><input disabled="" type="checkbox"/> Does each step get the minimum relevant evidence, not the maximum available text?</li> <li><input disabled="" type="checkbox"/> Are context items structured, fresh, and source-labeled?</li> <li><input disabled="" type="checkbox"/> Are trusted instructions kept apart from untrusted retrieved content?</li> <li><input disabled="" type="checkbox"/> Do you compact history and keep only durable memory?</li> </ul> <h3 id="harness-engineering-checklist">Harness engineering checklist<a class="zola-anchor" href="#harness-engineering-checklist" aria-label="Anchor link for: harness-engineering-checklist" style="visibility: hidden;"></a> </h3> <ul> <li><input disabled="" type="checkbox"/> Are tool permissions, side effects, and approval rules enforced in code?</li> <li><input disabled="" type="checkbox"/> Do you have retries, timeouts, and fallback behavior?</li> <li><input disabled="" type="checkbox"/> Can you replay a run with prompt, context, tool outputs, and validations?</li> <li><input disabled="" type="checkbox"/> Do you have eval cases for ambiguity, tool failure, and malicious context?</li> </ul> <h2 id="the-takeaway">The takeaway<a class="zola-anchor" href="#the-takeaway" aria-label="Anchor link for: the-takeaway" style="visibility: hidden;"></a> </h2> <p>If you are serious about agents, stop treating everything as prompt engineering.</p> <p>The prompt is the instruction layer. The context decides whether the model reasons over the right evidence. The harness decides whether the thing behaves like software you can run on-call or a demo that wows once and pages you at 2am.</p> <p>The pattern that lasts:</p> <ul> <li>keep prompts clear</li> <li>keep context relevant</li> <li>keep the harness deterministic where it has to be</li> </ul> <p>Get those three right and the agent stops being a demo you babysit and starts being software you can hand a real action.</p> 2026-03-13T00:00:00+00:00 2026-03-13T00:00:00+00:00 https://rajnandan.com/posts/owasp-top-10-ai-vulnerabilities/ <p>If you are putting LLMs into production, the main risk is no longer whether the demo works. The main risk is whether the system can be manipulated, over-trusted, or over-permissioned in ways that create a real security incident. That is why the <a rel="noopener nofollow noreferrer" target="_blank" href="https://owasp.org/www-project-top-10-for-large-language-model-applications/">OWASP Top 10 for LLM Applications</a> matters. First published in 2023 and updated for 2025, it turns vague AI anxiety into a concrete operating model for defending systems that reason over data, call tools, and influence downstream workflows.</p> <p>The most important takeaway is simple: most AI incidents are not caused by one magical "AI exploit." They happen when ordinary security weaknesses meet LLM-specific behavior. A prompt becomes an instruction. A retrieved document becomes a hidden control channel. A model output gets treated like trusted code. A helpful assistant gets enough permissions to act like an admin.</p> <p>This post walks through the 2025 OWASP Top 10 for LLM applications and, more importantly, what to do about each item in practice.</p> <h2 id="the-10-risks-at-a-glance">The 10 Risks at a Glance<a class="zola-anchor" href="#the-10-risks-at-a-glance" aria-label="Anchor link for: the-10-risks-at-a-glance" style="visibility: hidden;"></a> </h2> <table><thead><tr><th>OWASP item</th><th>What goes wrong</th><th>What to do first</th></tr></thead><tbody> <tr><td><strong>LLM01 Prompt Injection</strong></td><td>The model treats attacker-controlled input as instructions</td><td>Add input and output filtering, isolate untrusted content, and test with adversarial prompts</td></tr> <tr><td><strong>LLM02 Sensitive Information Disclosure</strong></td><td>The system leaks secrets, customer data, or internal knowledge</td><td>Sanitize data, restrict access, and inspect responses before release</td></tr> <tr><td><strong>LLM03 Supply Chain</strong></td><td>Models, adapters, datasets, and platforms arrive from untrusted sources</td><td>Vet suppliers, verify provenance, scan dependencies, and patch aggressively</td></tr> <tr><td><strong>LLM04 Data and Model Poisoning</strong></td><td>Training, fine-tuning, or RAG data is tampered with</td><td>Track data lineage, enforce change control, and validate against trusted sources</td></tr> <tr><td><strong>LLM05 Improper Output Handling</strong></td><td>LLM output is executed or rendered without validation</td><td>Treat model output as untrusted input and apply context-aware encoding and validation</td></tr> <tr><td><strong>LLM06 Excessive Agency</strong></td><td>The assistant can call too many tools with too much power</td><td>Minimize tool scope, minimize permissions, and require approval for high-risk actions</td></tr> <tr><td><strong>LLM07 System Prompt Leakage</strong></td><td>Internal instructions reveal secrets or guardrail details</td><td>Keep secrets out of prompts and enforce controls outside the model</td></tr> <tr><td><strong>LLM08 Vector and Embedding Weaknesses</strong></td><td>RAG stores leak data or retrieve poisoned context</td><td>Use permission-aware retrieval, validate documents, and log retrieval activity</td></tr> <tr><td><strong>LLM09 Misinformation</strong></td><td>The system sounds right while being wrong</td><td>Ground answers, cross-check outputs, and keep humans in the loop for high-stakes decisions</td></tr> <tr><td><strong>LLM10 Unbounded Consumption</strong></td><td>Attackers drive cost, latency, or service failure</td><td>Add rate limits, quotas, timeouts, and anomaly detection</td></tr> </tbody></table> <h2 id="1-prompt-injection-is-still-the-front-door">1. Prompt Injection Is Still the Front Door<a class="zola-anchor" href="#1-prompt-injection-is-still-the-front-door" aria-label="Anchor link for: 1-prompt-injection-is-still-the-front-door" style="visibility: hidden;"></a> </h2> <p>Prompt injection remains the number one issue because LLMs are not good at separating trusted instructions from untrusted input. A user can type a malicious instruction directly, or the model can ingest it indirectly from a webpage, PDF, email, or knowledge base document.</p> <p>That distinction matters:</p> <ul> <li><strong>Direct prompt injection</strong> comes from the user's own input</li> <li><strong>Indirect prompt injection</strong> is buried in external content the model reads on the user's behalf</li> </ul> <p>In both cases, the model may ignore its original instructions and follow the attacker's agenda instead. That can lead to data leakage, unsafe guidance, or arbitrary action in connected systems.</p> <h3 id="how-to-reduce-prompt-injection-risk">How to reduce prompt injection risk<a class="zola-anchor" href="#how-to-reduce-prompt-injection-risk" aria-label="Anchor link for: how-to-reduce-prompt-injection-risk" style="visibility: hidden;"></a> </h3> <ul> <li>Keep system prompts explicit about role, scope, and refusal behavior</li> <li>Put an AI gateway or filtering layer in front of the model and on the response path</li> <li>Clearly separate trusted instructions from untrusted retrieved content</li> <li>Treat external documents as hostile until validated</li> <li>Pen test the application with adversarial prompts, including obfuscated or encoded variants</li> </ul> <p>System prompts help, but they are not enough by themselves. The durable control is an external enforcement layer that inspects both the request and the response.</p> <h2 id="2-sensitive-information-disclosure-is-an-ai-specific-data-breach">2. Sensitive Information Disclosure Is an AI-Specific Data Breach<a class="zola-anchor" href="#2-sensitive-information-disclosure-is-an-ai-specific-data-breach" aria-label="Anchor link for: 2-sensitive-information-disclosure-is-an-ai-specific-data-breach" style="visibility: hidden;"></a> </h2> <p>LLMs are often trained on, connected to, or augmented with valuable data: customer records, health information, internal financials, source code, product plans, and proprietary workflows. Once that data is inside the system, the risk is not just accidental disclosure. It is also extraction over time.</p> <p>One practical example is the <strong>model inversion</strong> or extraction pattern: an attacker repeatedly queries the system, records the responses, and slowly harvests information that should never have been exposed.</p> <h3 id="how-to-reduce-disclosure-risk">How to reduce disclosure risk<a class="zola-anchor" href="#how-to-reduce-disclosure-risk" aria-label="Anchor link for: how-to-reduce-disclosure-risk" style="visibility: hidden;"></a> </h3> <ul> <li>Sanitize training and retrieval data before it enters the model pipeline</li> <li>Mask, tokenize, or redact high-risk fields such as credentials, account numbers, and personal identifiers</li> <li>Apply least-privilege access controls to the model, the data sources, and the users</li> <li>Inspect outbound responses for sensitive patterns before returning them</li> <li>Publish clear data usage and retention rules so users know what not to submit</li> </ul> <p>This is where conventional security discipline matters. Encryption, access control, data minimization, and configuration hygiene still apply. AI does not replace them.</p> <h2 id="3-supply-chain-risk-is-bigger-than-dependencies">3. Supply Chain Risk Is Bigger Than Dependencies<a class="zola-anchor" href="#3-supply-chain-risk-is-bigger-than-dependencies" aria-label="Anchor link for: 3-supply-chain-risk-is-bigger-than-dependencies" style="visibility: hidden;"></a> </h2> <p>Traditional software teams already understand package risk. LLM systems expand that problem. Your AI supply chain now includes:</p> <ul> <li>base models</li> <li>fine-tuned models</li> <li>LoRA adapters</li> <li>embeddings and datasets</li> <li>inference frameworks</li> <li>plug-ins and tool integrations</li> <li>the infrastructure that runs all of the above</li> </ul> <p>The difficult part is scale. Teams often consume models and artifacts they cannot realistically inspect by hand, especially from open repositories.</p> <h3 id="how-to-reduce-supply-chain-risk">How to reduce supply chain risk<a class="zola-anchor" href="#how-to-reduce-supply-chain-risk" aria-label="Anchor link for: how-to-reduce-supply-chain-risk" style="visibility: hidden;"></a> </h3> <ul> <li>Only use models, datasets, and adapters from suppliers you can verify</li> <li>Check provenance, hashes, signatures, and release history</li> <li>Maintain an inventory of models, components, and licenses</li> <li>Scan dependencies and runtime platforms for known vulnerabilities</li> <li>Keep patching policies current for both software and model-serving infrastructure</li> <li>Red team third-party models in the context you will actually deploy</li> </ul> <p>The practical lesson is straightforward: do not treat model downloads like neutral content. Treat them like executable supply chain inputs.</p> <h2 id="4-data-and-model-poisoning-corrupts-the-system-at-the-source">4. Data and Model Poisoning Corrupts the System at the Source<a class="zola-anchor" href="#4-data-and-model-poisoning-corrupts-the-system-at-the-source" aria-label="Anchor link for: 4-data-and-model-poisoning-corrupts-the-system-at-the-source" style="visibility: hidden;"></a> </h2> <p>LLMs are only as trustworthy as the data and artifacts used to shape them. If training data, fine-tuning inputs, or retrieval sources are subtly manipulated, the resulting system may still appear normal while producing biased, unsafe, or strategically wrong answers.</p> <p>Poisoning is especially dangerous because it can be quiet. A small amount of malicious content can alter outputs only when a specific trigger appears. That makes the system look healthy until the exact failure mode is activated.</p> <h3 id="how-to-reduce-poisoning-risk">How to reduce poisoning risk<a class="zola-anchor" href="#how-to-reduce-poisoning-risk" aria-label="Anchor link for: how-to-reduce-poisoning-risk" style="visibility: hidden;"></a> </h3> <ul> <li>Track where data came from and how it changed over time</li> <li>Require approval and change control for updates to models and knowledge sources</li> <li>Validate outputs against trusted references, especially in regulated domains</li> <li>Use anomaly detection and adversarial testing on training and retrieval pipelines</li> <li>Restrict who can modify model weights, fine-tuning data, and RAG corpora</li> </ul> <p>If your team uses Retrieval-Augmented Generation, remember that the RAG corpus is part of the attack surface. Poisoned documents can become poisoned answers.</p> <h2 id="5-improper-output-handling-turns-a-bad-answer-into-a-real-exploit">5. Improper Output Handling Turns a Bad Answer Into a Real Exploit<a class="zola-anchor" href="#5-improper-output-handling-turns-a-bad-answer-into-a-real-exploit" aria-label="Anchor link for: 5-improper-output-handling-turns-a-bad-answer-into-a-real-exploit" style="visibility: hidden;"></a> </h2> <p>A common design mistake is to treat model output as safe because it came from "our assistant." That is backwards. The output of an LLM should be handled the same way you would handle untrusted user input.</p> <p>This matters when model output is:</p> <ul> <li>rendered in a browser</li> <li>sent to an API</li> <li>converted into SQL</li> <li>written into shell commands</li> <li>inserted into file paths</li> <li>passed into automation tools</li> </ul> <p>Without validation, you can turn prompt injection into cross-site scripting, SQL injection, remote code execution, or destructive backend actions.</p> <h3 id="how-to-reduce-output-handling-risk">How to reduce output handling risk<a class="zola-anchor" href="#how-to-reduce-output-handling-risk" aria-label="Anchor link for: how-to-reduce-output-handling-risk" style="visibility: hidden;"></a> </h3> <ul> <li>Treat every model response as untrusted data</li> <li>Validate structure and schema before downstream use</li> <li>Use parameterized queries for database operations</li> <li>Apply context-aware output encoding for HTML, Markdown, and JavaScript</li> <li>Enforce content security policies where browser rendering is involved</li> <li>Log anomalous outputs and block unsafe execution paths</li> </ul> <p>The safe default is simple: never let raw LLM output execute anything important.</p> <h2 id="6-excessive-agency-makes-small-errors-catastrophic">6. Excessive Agency Makes Small Errors Catastrophic<a class="zola-anchor" href="#6-excessive-agency-makes-small-errors-catastrophic" aria-label="Anchor link for: 6-excessive-agency-makes-small-errors-catastrophic" style="visibility: hidden;"></a> </h2> <p>LLM systems become more dangerous when they can act, not just answer. The issue is not only whether the model hallucinates. It is also whether the model has enough power to turn a bad decision into a business, financial, or safety event.</p> <p>Excessive agency typically appears in three forms:</p> <ol> <li><strong>Excessive functionality</strong> - the model can call tools it does not need</li> <li><strong>Excessive permissions</strong> - the connected tools can do far more than the task requires</li> <li><strong>Excessive autonomy</strong> - the system performs high-impact actions without human approval</li> </ol> <h3 id="how-to-reduce-agency-risk">How to reduce agency risk<a class="zola-anchor" href="#how-to-reduce-agency-risk" aria-label="Anchor link for: how-to-reduce-agency-risk" style="visibility: hidden;"></a> </h3> <ul> <li>Offer only the minimum set of tools to the model</li> <li>Replace open-ended tools with narrow task-specific operations</li> <li>Run tool actions in the user's real authorization context</li> <li>Enforce authorization in downstream systems, not just in prompts</li> <li>Require human confirmation for destructive or sensitive actions</li> <li>Rate limit tool execution so one bad chain of reasoning cannot spiral quickly</li> </ul> <p>The more powerful the agent, the more deterministic the surrounding controls must be.</p> <h2 id="7-system-prompts-are-not-a-secret-store">7. System Prompts Are Not a Secret Store<a class="zola-anchor" href="#7-system-prompts-are-not-a-secret-store" aria-label="Anchor link for: 7-system-prompts-are-not-a-secret-store" style="visibility: hidden;"></a> </h2> <p>Many teams assume the system prompt is hidden and therefore safe. It is not. Attackers can often infer, extract, or approximate prompt contents simply by interacting with the model long enough.</p> <p>The real problem is not that the prompt text becomes visible. The real problem is storing secrets or policy-critical logic in a place the model can leak or reinterpret.</p> <h3 id="how-to-reduce-system-prompt-leakage-risk">How to reduce system prompt leakage risk<a class="zola-anchor" href="#how-to-reduce-system-prompt-leakage-risk" aria-label="Anchor link for: how-to-reduce-system-prompt-leakage-risk" style="visibility: hidden;"></a> </h3> <ul> <li>Never place credentials, tokens, connection strings, or private architecture details in prompts</li> <li>Keep permission checks and policy enforcement outside the model</li> <li>Use prompts for guidance, not as your primary security boundary</li> <li>Add output inspection that detects attempts to reveal internal instructions</li> </ul> <p>If a control would fail the moment the prompt leaks, it was never a strong control.</p> <h2 id="8-vector-and-embedding-weaknesses-expand-the-rag-attack-surface">8. Vector and Embedding Weaknesses Expand the RAG Attack Surface<a class="zola-anchor" href="#8-vector-and-embedding-weaknesses-expand-the-rag-attack-surface" aria-label="Anchor link for: 8-vector-and-embedding-weaknesses-expand-the-rag-attack-surface" style="visibility: hidden;"></a> </h2> <p>RAG improves relevance, but it also adds a new security boundary: embeddings, vector stores, retrieval logic, and document ingestion pipelines.</p> <p>That creates several practical risks:</p> <ul> <li>unauthorized retrieval across tenants or user classes</li> <li>poisoned documents that steer model behavior</li> <li>embedding inversion attacks that recover source information</li> <li>conflicting or stale context that degrades answer quality</li> </ul> <h3 id="how-to-reduce-vector-and-embedding-risk">How to reduce vector and embedding risk<a class="zola-anchor" href="#how-to-reduce-vector-and-embedding-risk" aria-label="Anchor link for: how-to-reduce-vector-and-embedding-risk" style="visibility: hidden;"></a> </h3> <ul> <li>Use permission-aware retrieval with strict logical separation between tenants and data classes</li> <li>Validate and classify documents before they enter the knowledge base</li> <li>Detect hidden or obfuscated content in uploaded files</li> <li>Log retrieval events so suspicious access patterns are visible</li> <li>Review how new retrieved context changes model behavior, not just factual accuracy</li> </ul> <p>RAG is not just a relevance feature. It is a security-sensitive data pipeline.</p> <h2 id="9-misinformation-is-a-security-problem-not-just-a-quality-problem">9. Misinformation Is a Security Problem, Not Just a Quality Problem<a class="zola-anchor" href="#9-misinformation-is-a-security-problem-not-just-a-quality-problem" aria-label="Anchor link for: 9-misinformation-is-a-security-problem-not-just-a-quality-problem" style="visibility: hidden;"></a> </h2> <p>LLMs can produce answers that sound credible while being false, unsupported, or dangerously incomplete. In low-stakes use cases that may be embarrassing. In legal, medical, financial, or security-sensitive workflows, it becomes liability.</p> <p>Misinformation often comes from:</p> <ul> <li>hallucinations</li> <li>incomplete or conflicting context</li> <li>poisoned data</li> <li>overreliance by users who assume the system is authoritative</li> </ul> <h3 id="how-to-reduce-misinformation-risk">How to reduce misinformation risk<a class="zola-anchor" href="#how-to-reduce-misinformation-risk" aria-label="Anchor link for: how-to-reduce-misinformation-risk" style="visibility: hidden;"></a> </h3> <ul> <li>Ground responses in trusted sources through retrieval and citation patterns</li> <li>Add automatic validation for critical outputs</li> <li>Require human review in high-stakes domains</li> <li>Design the interface to show uncertainty, limitations, and provenance</li> <li>Train users to verify important answers instead of accepting fluent text at face value</li> </ul> <p>The goal is not to eliminate every wrong answer. The goal is to stop wrong answers from quietly becoming decisions.</p> <h2 id="10-unbounded-consumption-leads-to-denial-of-service-and-denial-of-wallet">10. Unbounded Consumption Leads to Denial of Service and Denial of Wallet<a class="zola-anchor" href="#10-unbounded-consumption-leads-to-denial-of-service-and-denial-of-wallet" aria-label="Anchor link for: 10-unbounded-consumption-leads-to-denial-of-service-and-denial-of-wallet" style="visibility: hidden;"></a> </h2> <p>LLMs are resource-intensive by default. That makes them vulnerable to abuse by oversized inputs, repeated requests, complex prompts, and extraction-style querying. The result may look like classic denial of service, but in AI systems it often shows up first as a cost problem.</p> <p>This is why teams now use the phrase <strong>denial of wallet</strong>. An attacker may not need to crash the system. It may be enough to make it expensive enough to hurt.</p> <h3 id="how-to-reduce-consumption-risk">How to reduce consumption risk<a class="zola-anchor" href="#how-to-reduce-consumption-risk" aria-label="Anchor link for: how-to-reduce-consumption-risk" style="visibility: hidden;"></a> </h3> <ul> <li>Limit request size, context length, and queued actions</li> <li>Apply rate limits, quotas, and per-user budgets</li> <li>Set timeouts and throttles for expensive operations</li> <li>Monitor token usage, latency spikes, and abnormal request patterns</li> <li>Degrade gracefully under load rather than failing completely</li> <li>Watch for repeated querying patterns that resemble extraction attempts</li> </ul> <p>Cost controls are security controls when inference is expensive.</p> <h2 id="a-practical-defense-stack-for-llm-applications">A Practical Defense Stack for LLM Applications<a class="zola-anchor" href="#a-practical-defense-stack-for-llm-applications" aria-label="Anchor link for: a-practical-defense-stack-for-llm-applications" style="visibility: hidden;"></a> </h2> <p>If you want one implementation-oriented view, the controls below do the most work across the full Top 10:</p> <h3 id="1-ai-gateway-or-firewall">1. AI gateway or firewall<a class="zola-anchor" href="#1-ai-gateway-or-firewall" aria-label="Anchor link for: 1-ai-gateway-or-firewall" style="visibility: hidden;"></a> </h3> <p>Inspect prompts going in and responses coming out. Block obvious prompt injection, redact sensitive data, and enforce policy before the model or the user sees anything.</p> <h3 id="2-least-privilege-architecture">2. Least-privilege architecture<a class="zola-anchor" href="#2-least-privilege-architecture" aria-label="Anchor link for: 2-least-privilege-architecture" style="visibility: hidden;"></a> </h3> <p>Limit what the model can access, what its tools can do, and what downstream identities are allowed to perform. Keep authorization outside the model.</p> <h3 id="3-data-hygiene-and-provenance">3. Data hygiene and provenance<a class="zola-anchor" href="#3-data-hygiene-and-provenance" aria-label="Anchor link for: 3-data-hygiene-and-provenance" style="visibility: hidden;"></a> </h3> <p>Know where training, fine-tuning, and RAG data came from. Validate it, version it, and control changes to it.</p> <h3 id="4-output-validation">4. Output validation<a class="zola-anchor" href="#4-output-validation" aria-label="Anchor link for: 4-output-validation" style="visibility: hidden;"></a> </h3> <p>Do not let the model write raw SQL, HTML, shell commands, or API parameters into real systems without deterministic validation.</p> <h3 id="5-human-approval-for-high-risk-actions">5. Human approval for high-risk actions<a class="zola-anchor" href="#5-human-approval-for-high-risk-actions" aria-label="Anchor link for: 5-human-approval-for-high-risk-actions" style="visibility: hidden;"></a> </h3> <p>If the action moves money, changes records, deletes data, sends external communications, or impacts health and safety, require human review.</p> <h3 id="6-monitoring-logging-and-anomaly-detection">6. Monitoring, logging, and anomaly detection<a class="zola-anchor" href="#6-monitoring-logging-and-anomaly-detection" aria-label="Anchor link for: 6-monitoring-logging-and-anomaly-detection" style="visibility: hidden;"></a> </h3> <p>Log prompts, retrieval events, tool calls, and outcomes with enough structure to replay incidents and detect abuse patterns.</p> <h3 id="7-adversarial-testing">7. Adversarial testing<a class="zola-anchor" href="#7-adversarial-testing" aria-label="Anchor link for: 7-adversarial-testing" style="visibility: hidden;"></a> </h3> <p>Regularly test with direct injections, indirect injections, poisoned content, malicious documents, and tool abuse scenarios. If you do not test these paths, attackers will.</p> <h2 id="a-production-checklist">A Production Checklist<a class="zola-anchor" href="#a-production-checklist" aria-label="Anchor link for: a-production-checklist" style="visibility: hidden;"></a> </h2> <ul> <li><input disabled="" type="checkbox"/> Sanitize sensitive data before training, fine-tuning, or retrieval ingestion</li> <li><input disabled="" type="checkbox"/> Treat prompts, retrieved content, and model output as untrusted by default</li> <li><input disabled="" type="checkbox"/> Put approval gates around destructive or externally visible actions</li> <li><input disabled="" type="checkbox"/> Enforce least privilege for tools, plug-ins, APIs, and vector stores</li> <li><input disabled="" type="checkbox"/> Verify model, adapter, and dataset provenance before deployment</li> <li><input disabled="" type="checkbox"/> Validate output before passing it to browsers, databases, shells, or business systems</li> <li><input disabled="" type="checkbox"/> Apply quotas, rate limits, and timeouts to control denial-of-wallet risk</li> <li><input disabled="" type="checkbox"/> Log enough detail to trace incidents across prompt, retrieval, tool, and response steps</li> <li><input disabled="" type="checkbox"/> Red team the application, not just the base model</li> </ul> <h2 id="further-reading">Further Reading<a class="zola-anchor" href="#further-reading" aria-label="Anchor link for: further-reading" style="visibility: hidden;"></a> </h2> <p>If you want the original OWASP write-ups behind this summary, start here:</p> <ul> <li><a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/raw/020595761a4b7b0c3f9cf01a0457b78f9f1e7f9c/2_0_vulns/LLM01_PromptInjection.md">LLM01: Prompt Injection</a></li> <li><a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/raw/020595761a4b7b0c3f9cf01a0457b78f9f1e7f9c/2_0_vulns/LLM02_SensitiveInformationDisclosure.md">LLM02: Sensitive Information Disclosure</a></li> <li><a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/raw/020595761a4b7b0c3f9cf01a0457b78f9f1e7f9c/2_0_vulns/LLM03_SupplyChain.md">LLM03: Supply Chain</a></li> <li><a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/raw/020595761a4b7b0c3f9cf01a0457b78f9f1e7f9c/2_0_vulns/LLM04_DataModelPoisoning.md">LLM04: Data and Model Poisoning</a></li> <li><a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/raw/020595761a4b7b0c3f9cf01a0457b78f9f1e7f9c/2_0_vulns/LLM05_ImproperOutputHandling.md">LLM05: Improper Output Handling</a></li> <li><a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/raw/020595761a4b7b0c3f9cf01a0457b78f9f1e7f9c/2_0_vulns/LLM06_ExcessiveAgency.md">LLM06: Excessive Agency</a></li> <li><a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/raw/020595761a4b7b0c3f9cf01a0457b78f9f1e7f9c/2_0_vulns/LLM07_SystemPromptLeakage.md">LLM07: System Prompt Leakage</a></li> <li><a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/raw/020595761a4b7b0c3f9cf01a0457b78f9f1e7f9c/2_0_vulns/LLM08_VectorAndEmbeddingWeaknesses.md">LLM08: Vector and Embedding Weaknesses</a></li> <li><a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/raw/020595761a4b7b0c3f9cf01a0457b78f9f1e7f9c/2_0_vulns/LLM09_Misinformation.md">LLM09: Misinformation</a></li> <li><a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/raw/020595761a4b7b0c3f9cf01a0457b78f9f1e7f9c/2_0_vulns/LLM10_UnboundedConsumption.md">LLM10: Unbounded Consumption</a></li> </ul> <h2 id="the-bottom-line">The Bottom Line<a class="zola-anchor" href="#the-bottom-line" aria-label="Anchor link for: the-bottom-line" style="visibility: hidden;"></a> </h2> <p>The OWASP Top 10 for LLMs is useful because it reframes AI security as engineering discipline. Prompt injection, data leakage, poisoned retrieval, untrusted output, and overpowered agents are not abstract research curiosities. They are production failure modes.</p> <p>The teams that operate LLMs safely will be the teams that treat them neither as magic nor as ordinary software. They will treat them as probabilistic systems wrapped in deterministic controls.</p> <p>That is the right mental model for building AI that is useful, secure, and still under your control.</p> 2026-03-10T00:00:00+00:00 2026-03-10T00:00:00+00:00 https://rajnandan.com/posts/p-vs-np-for-everyone/ <p>Most people first hear <strong>P vs NP</strong> and immediately think, "This sounds important, but I have no idea what it means."</p> <p>That reaction is normal.</p> <p>Here is the whole idea in one sentence:</p> <blockquote> <p>Some problems are easy to solve. Some problems are hard to solve but easy to check once somebody hands you an answer. P vs NP asks whether those two groups are actually the same group.</p> </blockquote> <p>This matters because many real problems look like this:</p> <ul> <li>finding the best delivery route</li> <li>scheduling workers</li> <li>packing boxes into a truck</li> <li>designing computer chips</li> <li>deciding how to place ads, servers, or warehouses</li> <li>solving puzzles such as Sudoku</li> </ul> <p>If every "easy to check" problem were also "easy to solve," a huge part of planning, optimization, and cryptography would look different.</p> <p>This article explains the idea in the simplest way possible. No math degree required.</p> <h2 id="the-tiny-mental-model">The Tiny Mental Model<a class="zola-anchor" href="#the-tiny-mental-model" aria-label="Anchor link for: the-tiny-mental-model" style="visibility: hidden;"></a> </h2> <p>Imagine four kinds of tasks:</p> <ol> <li><strong>P</strong>: easy to solve</li> <li><strong>NP</strong>: maybe hard to solve, but easy to check</li> <li><strong>NP-hard</strong>: at least as hard as the hardest NP problems</li> <li><strong>NP-complete</strong>: the hardest problems inside NP</li> </ol> <p>If that still feels abstract, do not worry. We will build it with toy examples.</p> <h2 id="start-with-a-toy-example">Start With a Toy Example<a class="zola-anchor" href="#start-with-a-toy-example" aria-label="Anchor link for: start-with-a-toy-example" style="visibility: hidden;"></a> </h2> <p>Imagine you have a box of letter magnets.</p> <p>You want to know whether these letters can be arranged to make the word:</p> <p><strong>APPLE</strong></p> <h3 id="version-1-the-letters-are-already-in-order">Version 1: The letters are already in order<a class="zola-anchor" href="#version-1-the-letters-are-already-in-order" aria-label="Anchor link for: version-1-the-letters-are-already-in-order" style="visibility: hidden;"></a> </h3> <p>You see:</p> <p><strong>A P P L E</strong></p> <p>That is easy. You can read it and answer right away.</p> <p>This feels like a <strong>P</strong> problem.</p> <h3 id="version-2-the-letters-are-all-mixed-up">Version 2: The letters are all mixed up<a class="zola-anchor" href="#version-2-the-letters-are-all-mixed-up" aria-label="Anchor link for: version-2-the-letters-are-all-mixed-up" style="visibility: hidden;"></a> </h3> <p>You see:</p> <p><strong>L P A E P</strong></p> <p>Now the job is harder. You may need to rearrange them and think for a moment.</p> <p>But if your friend says, "The answer is APPLE," checking is easy. You just compare the letters.</p> <p>This is the basic feeling of <strong>NP</strong>.</p> <ul> <li>Finding the answer may be hard.</li> <li>Checking a proposed answer is easy.</li> </ul> <p>That is the heart of the whole story.</p> <h2 id="what-does-p-mean">What Does P Mean?<a class="zola-anchor" href="#what-does-p-mean" aria-label="Anchor link for: what-does-p-mean" style="visibility: hidden;"></a> </h2> <p><strong>P</strong> stands for problems that a computer can solve efficiently.</p> <p>In plain English, P means:</p> <ul> <li>the problem has a method that scales reasonably well</li> <li>when the input gets bigger, the work grows, but not explosively</li> <li>we know a practical step-by-step way to get the answer</li> </ul> <h3 id="child-sized-example">Child-sized example<a class="zola-anchor" href="#child-sized-example" aria-label="Anchor link for: child-sized-example" style="visibility: hidden;"></a> </h3> <p>You have 10 toy cars in a line, from smallest to biggest. You want to know if the blue car is in the line.</p> <p>You can scan the line and check one by one.</p> <p>That is a straightforward problem. Even if the line gets longer, the process stays sensible.</p> <h3 id="more-everyday-examples-of-p-like-tasks">More everyday examples of P-like tasks<a class="zola-anchor" href="#more-everyday-examples-of-p-like-tasks" aria-label="Anchor link for: more-everyday-examples-of-p-like-tasks" style="visibility: hidden;"></a> </h3> <ul> <li>alphabetizing a short list of names</li> <li>checking whether a number is even</li> <li>finding the shortest path on a map when the map has normal road weights and we use known efficient algorithms</li> <li>checking whether one word appears in a paragraph</li> </ul> <p>The important feeling is this: <strong>we know how to get the answer without trying a ridiculous number of possibilities</strong>.</p> <h2 id="what-does-np-mean">What Does NP Mean?<a class="zola-anchor" href="#what-does-np-mean" aria-label="Anchor link for: what-does-np-mean" style="visibility: hidden;"></a> </h2> <p><strong>NP</strong> stands for a set of problems where a proposed answer can be checked efficiently.</p> <p>For non-technical readers, this is the best way to remember it:</p> <blockquote> <p><strong>NP = hard to figure out, easy to verify.</strong></p> </blockquote> <p>That is not the full historical definition, but it is the most useful one for understanding the idea.</p> <h3 id="child-sized-example-a-jigsaw-puzzle">Child-sized example: a jigsaw puzzle<a class="zola-anchor" href="#child-sized-example-a-jigsaw-puzzle" aria-label="Anchor link for: child-sized-example-a-jigsaw-puzzle" style="visibility: hidden;"></a> </h3> <p>Suppose I dump a 500-piece puzzle on the floor.</p> <ul> <li><strong>Solving it from scratch</strong> can take a long time.</li> <li><strong>Checking a finished puzzle</strong> is much easier. You can look and see whether every piece fits and the picture is correct.</li> </ul> <p>That is an NP-style situation.</p> <h3 id="another-example-sudoku">Another example: Sudoku<a class="zola-anchor" href="#another-example-sudoku" aria-label="Anchor link for: another-example-sudoku" style="visibility: hidden;"></a> </h3> <ul> <li>Filling a Sudoku grid can be hard.</li> <li>Checking whether a completed grid obeys the rules is much easier.</li> </ul> <p>Again, that is the feeling of NP.</p> <h2 id="so-what-is-the-actual-p-vs-np-question">So What Is the Actual P vs NP Question?<a class="zola-anchor" href="#so-what-is-the-actual-p-vs-np-question" aria-label="Anchor link for: so-what-is-the-actual-p-vs-np-question" style="visibility: hidden;"></a> </h2> <p>Now we can ask the big question.</p> <p>If a problem is easy to <strong>check</strong>, is it also easy to <strong>solve</strong>?</p> <p>That is what <strong>P vs NP</strong> asks.</p> <ul> <li>If <strong>P = NP</strong>, then every problem with quickly checkable answers also has a quickly findable answer.</li> <li>If <strong>P ≠ NP</strong>, then there are some problems where checking is easy, but finding remains fundamentally hard.</li> </ul> <p>Most computer scientists believe <strong>P ≠ NP</strong>, but no one has proved it.</p> <p>As of March 2026, the problem is still unsolved and remains one of the Clay Mathematics Institute's Millennium Prize Problems, with a $1 million prize for a correct solution. See Clay's overview of <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.claymath.org/millennium/p-vs-np/">P vs NP</a> and the broader <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.claymath.org/millennium-problems/">Millennium Prize Problems</a>.</p> <h2 id="why-easy-to-check-is-not-the-same-as-easy-to-find">Why "Easy to Check" Is Not the Same as "Easy to Find"<a class="zola-anchor" href="#why-easy-to-check-is-not-the-same-as-easy-to-find" aria-label="Anchor link for: why-easy-to-check-is-not-the-same-as-easy-to-find" style="visibility: hidden;"></a> </h2> <p>This is where many people get stuck, so let us slow down.</p> <p>Imagine I ask:</p> <blockquote> <p>Can you guess my five-digit lock code?</p> </blockquote> <p>That can take a long time if you have no clue.</p> <p>Now imagine I say:</p> <blockquote> <p>I think the code is 38142.</p> </blockquote> <p>Checking is easy. You type it once.</p> <p>So:</p> <ul> <li><strong>finding</strong> can be hard</li> <li><strong>checking</strong> can be easy</li> </ul> <p>That gap is the whole reason NP exists as a separate idea.</p> <h2 id="what-is-np-complete">What Is NP-Complete?<a class="zola-anchor" href="#what-is-np-complete" aria-label="Anchor link for: what-is-np-complete" style="visibility: hidden;"></a> </h2> <p>Now we get to the famous phrase.</p> <p>A problem is <strong>NP-complete</strong> if both of these are true:</p> <ol> <li>it is in <strong>NP</strong>, so a proposed answer can be checked efficiently</li> <li>it is as hard as any problem in NP</li> </ol> <p>That second part is the big one.</p> <p>If you found an efficient method for <strong>one</strong> NP-complete problem, you would effectively unlock efficient methods for <strong>all</strong> NP problems.</p> <p>That is why NP-complete problems are such a big deal.</p> <h3 id="a-simple-way-to-picture-np-complete">A simple way to picture NP-complete<a class="zola-anchor" href="#a-simple-way-to-picture-np-complete" aria-label="Anchor link for: a-simple-way-to-picture-np-complete" style="visibility: hidden;"></a> </h3> <p>Think of NP as a mountain range.</p> <ul> <li>Some hills may be easier than others.</li> <li>NP-complete problems sit among the tallest, toughest peaks.</li> </ul> <p>They are the benchmark hard problems inside NP.</p> <h3 id="famous-example-sat">Famous example: SAT<a class="zola-anchor" href="#famous-example-sat" aria-label="Anchor link for: famous-example-sat" style="visibility: hidden;"></a> </h3> <p>One of the most famous NP-complete problems is <strong>SAT</strong>, short for the Boolean satisfiability problem.</p> <p>Very roughly, SAT asks:</p> <blockquote> <p>Can I assign true/false values to variables so that a logical statement becomes true?</p> </blockquote> <p>That may sound niche, but SAT matters because many planning and constraint problems can be translated into it.</p> <p>The reason SAT is historically important is that Stephen Cook's 1971 paper established the first NP-complete problem, and SAT became the classic starting point for many later NP-completeness proofs. For a readable source, see MIT OpenCourseWare's notes on the <a rel="noopener nofollow noreferrer" target="_blank" href="https://ocw.mit.edu/courses/18-404j-theory-of-computation-fall-2020/3a8e803c31ff396ad5338677489dc7ea_6Az1gtDRaAU.pdf">Cook-Levin theorem</a>.</p> <h2 id="what-is-np-hard">What Is NP-Hard?<a class="zola-anchor" href="#what-is-np-hard" aria-label="Anchor link for: what-is-np-hard" style="visibility: hidden;"></a> </h2> <p>Now let us make one important distinction.</p> <p>A problem is <strong>NP-hard</strong> if it is at least as hard as the hardest problems in NP.</p> <p>But unlike NP-complete problems, an NP-hard problem does <strong>not</strong> have to be in NP.</p> <p>That means:</p> <ul> <li>it may not be a yes-or-no question</li> <li>it may not have an answer that is easy to check</li> <li>it may be even broader or messier</li> </ul> <h3 id="easy-example-shortest-trip-vs-trip-under-a-limit">Easy example: shortest trip vs trip under a limit<a class="zola-anchor" href="#easy-example-shortest-trip-vs-trip-under-a-limit" aria-label="Anchor link for: easy-example-shortest-trip-vs-trip-under-a-limit" style="visibility: hidden;"></a> </h3> <p>Imagine you need to visit 10 houses and come back home.</p> <h4 id="question-a">Question A<a class="zola-anchor" href="#question-a" aria-label="Anchor link for: question-a" style="visibility: hidden;"></a> </h4> <blockquote> <p>Is there a route that visits every house and takes 30 minutes or less?</p> </blockquote> <p>That is a <strong>yes/no</strong> question.</p> <p>This kind of decision version is the kind of thing complexity theory talks about, and versions of this question can be <strong>NP-complete</strong>.</p> <h4 id="question-b">Question B<a class="zola-anchor" href="#question-b" aria-label="Anchor link for: question-b" style="visibility: hidden;"></a> </h4> <blockquote> <p>What is the absolute shortest possible route?</p> </blockquote> <p>That is an optimization problem.</p> <p>This kind of version is usually described as <strong>NP-hard</strong>.</p> <p>Why? Because if you could solve the optimization version quickly, you could also answer the yes/no version quickly.</p> <p>So the optimization version is at least as hard.</p> <h2 id="np-hard-vs-np-complete-in-one-breath">NP-Hard vs NP-Complete in One Breath<a class="zola-anchor" href="#np-hard-vs-np-complete-in-one-breath" aria-label="Anchor link for: np-hard-vs-np-complete-in-one-breath" style="visibility: hidden;"></a> </h2> <p>If you want the shortest memory trick, use this:</p> <ul> <li><strong>NP</strong>: easy to check</li> <li><strong>NP-complete</strong>: hardest problems that are still easy to check</li> <li><strong>NP-hard</strong>: at least that hard, but not necessarily easy to check</li> </ul> <p>Or even shorter:</p> <ul> <li><strong>NP-complete = inside NP and very hard</strong></li> <li><strong>NP-hard = at least that hard, maybe outside NP</strong></li> </ul> <h2 id="a-simple-table">A Simple Table<a class="zola-anchor" href="#a-simple-table" aria-label="Anchor link for: a-simple-table" style="visibility: hidden;"></a> </h2> <table><thead><tr><th>Idea</th><th>What it means in plain English</th><th>Easy to check?</th><th>Easy to solve?</th></tr></thead><tbody> <tr><td><strong>P</strong></td><td>We know an efficient way to solve it</td><td>Yes</td><td>Yes</td></tr> <tr><td><strong>NP</strong></td><td>A proposed answer can be checked efficiently</td><td>Yes</td><td>Not known</td></tr> <tr><td><strong>NP-complete</strong></td><td>One of the hardest problems in NP</td><td>Yes</td><td>Not known</td></tr> <tr><td><strong>NP-hard</strong></td><td>At least as hard as NP-complete problems</td><td>Not always</td><td>Not known</td></tr> </tbody></table> <h2 id="why-do-people-care-so-much">Why Do People Care So Much?<a class="zola-anchor" href="#why-do-people-care-so-much" aria-label="Anchor link for: why-do-people-care-so-much" style="visibility: hidden;"></a> </h2> <p>Because these ideas are not just about puzzles.</p> <p>They shape how we think about real systems.</p> <h3 id="1-planning-and-logistics">1. Planning and logistics<a class="zola-anchor" href="#1-planning-and-logistics" aria-label="Anchor link for: 1-planning-and-logistics" style="visibility: hidden;"></a> </h3> <p>Companies want the best route, best schedule, best packing plan, best warehouse placement, and best use of machines.</p> <p>Many of these problems become brutally hard as the number of choices grows.</p> <h3 id="2-software-and-hardware-design">2. Software and hardware design<a class="zola-anchor" href="#2-software-and-hardware-design" aria-label="Anchor link for: 2-software-and-hardware-design" style="visibility: hidden;"></a> </h3> <p>Chip design, compiler work, verification, testing, and resource allocation often involve huge search spaces.</p> <h3 id="3-cryptography">3. Cryptography<a class="zola-anchor" href="#3-cryptography" aria-label="Anchor link for: 3-cryptography" style="visibility: hidden;"></a> </h3> <p>Modern security often depends on some tasks being hard to solve quickly.</p> <p>A proof that <strong>P = NP</strong> would not instantly break every security system overnight, but it would shake some of the assumptions people rely on when designing secure systems.</p> <h3 id="4-everyday-trade-offs">4. Everyday trade-offs<a class="zola-anchor" href="#4-everyday-trade-offs" aria-label="Anchor link for: 4-everyday-trade-offs" style="visibility: hidden;"></a> </h3> <p>When a problem is probably hard in the NP sense, engineers stop chasing perfect answers for massive inputs and start using:</p> <ul> <li>approximations</li> <li>heuristics</li> <li>clever shortcuts</li> <li>special-case algorithms</li> <li>good-enough answers</li> </ul> <p>That is actually an important practical lesson.</p> <p>Sometimes the smartest move is not "find the perfect answer." It is "find a very good answer fast enough to be useful."</p> <h2 id="the-biggest-misunderstanding">The Biggest Misunderstanding<a class="zola-anchor" href="#the-biggest-misunderstanding" aria-label="Anchor link for: the-biggest-misunderstanding" style="visibility: hidden;"></a> </h2> <p>Many people hear "NP" and think it means "not possible" or "non-polynomial."</p> <p>It does not.</p> <p>For beginners, just remember this:</p> <ul> <li><strong>P</strong> is the group of efficiently solvable problems</li> <li><strong>NP</strong> is the group of efficiently checkable problems</li> </ul> <p>And yes, every problem in <strong>P</strong> is also in <strong>NP</strong>, because if you can solve something quickly, then you can certainly check an answer quickly too.</p> <p>So the relationship looks like this:</p> <ul> <li><strong>P</strong> sits inside <strong>NP</strong></li> <li><strong>NP-complete</strong> sits inside <strong>NP</strong></li> <li><strong>NP-hard</strong> overlaps with NP-complete but can also extend beyond NP</li> </ul> <h2 id="if-this-is-so-important-why-has-nobody-solved-it">If This Is So Important, Why Has Nobody Solved It?<a class="zola-anchor" href="#if-this-is-so-important-why-has-nobody-solved-it" aria-label="Anchor link for: if-this-is-so-important-why-has-nobody-solved-it" style="visibility: hidden;"></a> </h2> <p>Because proving things about <em>all possible algorithms</em> is extremely hard.</p> <p>It is one thing to say, "I tried a lot of approaches and none worked." It is another thing to prove, forever, that no efficient method can exist.</p> <p>That is why P vs NP is not just a programming problem. It is a deep mathematical question about the nature of computation itself.</p> <h2 id="the-latest-status-and-research-without-hype">The Latest Status and Research, Without Hype<a class="zola-anchor" href="#the-latest-status-and-research-without-hype" aria-label="Anchor link for: the-latest-status-and-research-without-hype" style="visibility: hidden;"></a> </h2> <p>Here is the honest update.</p> <h3 id="1-the-main-question-is-still-open">1. The main question is still open<a class="zola-anchor" href="#1-the-main-question-is-still-open" aria-label="Anchor link for: 1-the-main-question-is-still-open" style="visibility: hidden;"></a> </h3> <p>As of March 2026, there is still no accepted proof that <strong>P = NP</strong> or that <strong>P ≠ NP</strong>.</p> <p>The big headline is actually the lack of a final headline.</p> <h3 id="2-research-is-still-very-active">2. Research is still very active<a class="zola-anchor" href="#2-research-is-still-very-active" aria-label="Anchor link for: 2-research-is-still-very-active" style="visibility: hidden;"></a> </h3> <p>Researchers have not stopped. They continue to work on nearby questions and useful partial progress, including:</p> <ul> <li><strong>complexity lower bounds</strong>, which try to prove that some problems really do require large amounts of computation</li> <li><strong>proof complexity</strong>, which studies how hard it is to prove certain statements inside formal systems</li> <li><strong>meta-complexity</strong>, which studies the complexity of questions about complexity itself, such as how hard it is to describe or compress computations</li> </ul> <p>If you want to see what active research looks like, two good high-level starting points are the Simons Institute program on <a rel="noopener nofollow noreferrer" target="_blank" href="https://simons.berkeley.edu/programs/Meta-Complexity2023">Meta-Complexity</a> and Clay Mathematics Institute's 2025 workshop on <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.claymath.org/events/p-vs-np-and-complexity-lower-bounds/">P vs NP and Complexity Lower Bounds</a>.</p> <h3 id="3-practical-progress-keeps-happening-even-without-a-final-proof">3. Practical progress keeps happening even without a final proof<a class="zola-anchor" href="#3-practical-progress-keeps-happening-even-without-a-final-proof" aria-label="Anchor link for: 3-practical-progress-keeps-happening-even-without-a-final-proof" style="visibility: hidden;"></a> </h3> <p>Even though the grand question is unresolved, researchers and engineers keep improving:</p> <ul> <li>SAT solvers</li> <li>optimization methods</li> <li>approximation algorithms</li> <li>special-purpose heuristics</li> <li>domain-specific tools for scheduling, routing, and verification</li> </ul> <p>So the theory question remains open, but the practical toolbox keeps getting better.</p> <p>That is why you sometimes hear two statements that are both true:</p> <ul> <li>"P vs NP is unsolved."</li> <li>"We are better than ever at solving many hard instances in practice."</li> </ul> <p>Those do not conflict.</p> <h2 id="the-five-year-old-version">The Five-Year-Old Version<a class="zola-anchor" href="#the-five-year-old-version" aria-label="Anchor link for: the-five-year-old-version" style="visibility: hidden;"></a> </h2> <p>If you want the shortest possible explanation, here it is.</p> <ul> <li><strong>P</strong> means: I can solve it quickly.</li> <li><strong>NP</strong> means: I might not solve it quickly, but if you give me an answer, I can check it quickly.</li> <li><strong>NP-hard</strong> means: this problem is at least as tough as the toughest checkable problems.</li> <li><strong>NP-complete</strong> means: this is one of the toughest checkable problems.</li> </ul> <p>And <strong>P vs NP</strong> asks:</p> <blockquote> <p>If checking is easy, is solving also easy?</p> </blockquote> <p>Nobody knows yet.</p> <h2 id="final-recap">Final Recap<a class="zola-anchor" href="#final-recap" aria-label="Anchor link for: final-recap" style="visibility: hidden;"></a> </h2> <p>Let us end with one clean picture.</p> <p>Imagine a giant room full of puzzles.</p> <ul> <li><strong>P puzzles</strong> are the ones you know how to solve efficiently.</li> <li><strong>NP puzzles</strong> are the ones where a finished answer is easy to inspect.</li> <li><strong>NP-complete puzzles</strong> are the toughest puzzles in that inspectable group.</li> <li><strong>NP-hard puzzles</strong> are at least that tough, and some are even outside that group.</li> </ul> <p>So the real mystery is simple to say and very hard to settle:</p> <blockquote> <p>Are all easy-to-check puzzles also easy-to-solve puzzles?</p> </blockquote> <p>That is P vs NP.</p> <h2 id="further-reading">Further Reading<a class="zola-anchor" href="#further-reading" aria-label="Anchor link for: further-reading" style="visibility: hidden;"></a> </h2> <ul> <li>Clay Mathematics Institute, <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.claymath.org/millennium/p-vs-np/">P vs NP</a></li> <li>Clay Mathematics Institute, <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.claymath.org/millennium-problems/">Millennium Prize Problems</a></li> <li>MIT OpenCourseWare, <a rel="noopener nofollow noreferrer" target="_blank" href="https://ocw.mit.edu/courses/18-404j-theory-of-computation-fall-2020/3a8e803c31ff396ad5338677489dc7ea_6Az1gtDRaAU.pdf">Cook-Levin Theorem</a></li> <li>Simons Institute, <a rel="noopener nofollow noreferrer" target="_blank" href="https://simons.berkeley.edu/programs/Meta-Complexity2023">Meta-Complexity</a></li> <li>Clay Mathematics Institute, <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.claymath.org/events/p-vs-np-and-complexity-lower-bounds/">P vs NP and Complexity Lower Bounds</a></li> </ul> 2026-03-09T00:00:00+00:00 2026-03-09T00:00:00+00:00 https://rajnandan.com/posts/how-a-screen-turns-electricity-into-motion/ <p>A screen looks effortless. Tap a button, drag a map, start a game, and the image seems to react instantly. But a display is not painting a continuous world. It is running a tightly choreographed loop: receive data, decide pixel values, push light toward your eyes, and repeat that cycle dozens or even hundreds of times each second.</p> <p>That loop matters because it shapes almost every digital experience we care about. Smooth scrolling, readable text, responsive games, battery life, eye comfort, and color accuracy all depend on how well a screen performs this job. Once you see the machinery behind the glass, a display stops feeling like a flat object and starts feeling like a live system.</p> <h2 id="the-screen-is-really-a-grid-of-tiny-light-decisions">The Screen Is Really a Grid of Tiny Light Decisions<a class="zola-anchor" href="#the-screen-is-really-a-grid-of-tiny-light-decisions" aria-label="Anchor link for: the-screen-is-really-a-grid-of-tiny-light-decisions" style="visibility: hidden;"></a> </h2> <p>At the lowest level, a screen is a grid. Each point in that grid is a pixel, and each pixel has one job: emit the right color at the right brightness at the right moment.</p> <p>That sounds simple until you multiply it by modern resolutions:</p> <ul> <li>1920 x 1080 means about 2.1 million pixels</li> <li>2560 x 1440 means about 3.7 million pixels</li> <li>3840 x 2160 means about 8.3 million pixels</li> </ul> <p>Now remember that each pixel is usually made of three subpixels: red, green, and blue. So a 4K display is not managing 8.3 million light sources. It is managing roughly 25 million subpixels, and it has to coordinate them continuously.</p> <p>Every image you see is just the result of millions of these tiny components being told, "be a little brighter," "go darker," "turn blue," or "stay off." A display is less like a canvas and more like a city of microscopic lamps following instructions with strict timing.</p> <h2 id="pixels-do-not-move-the-illusion-does">Pixels Do Not Move. The Illusion Does<a class="zola-anchor" href="#pixels-do-not-move-the-illusion-does" aria-label="Anchor link for: pixels-do-not-move-the-illusion-does" style="visibility: hidden;"></a> </h2> <p>The strangest thing about a screen is that motion is mostly a trick. The display never shows movement itself. It shows one still image, then another, then another, fast enough that your visual system stitches them together into motion.</p> <p>That is why frame rate matters so much.</p> <ul> <li>At 24 frames per second, motion can feel cinematic</li> <li>At 60 frames per second, interfaces generally feel fluid</li> <li>At 120 frames per second and above, motion can feel tighter and more immediate</li> </ul> <p>The improvement is not magic. The screen is simply giving your brain more frequent updates. The gap between one image and the next gets smaller, so movement appears more continuous and control feels more direct.</p> <p>This is also why poor frame pacing feels bad even when the average frame rate looks acceptable. If the timing between frames is uneven, motion stops feeling natural. Your eyes are good at spotting inconsistency, especially while scrolling text or tracking a moving object.</p> <h2 id="a-refresh-is-a-deadline">A Refresh Is a Deadline<a class="zola-anchor" href="#a-refresh-is-a-deadline" aria-label="Anchor link for: a-refresh-is-a-deadline" style="visibility: hidden;"></a> </h2> <p>People often talk about refresh rate as a feature, but it is more useful to think of it as a deadline.</p> <p>A 60 Hz display refreshes 60 times per second. That gives the system about 16.7 milliseconds to prepare each new frame. A 120 Hz display cuts that budget to roughly 8.3 milliseconds. A 144 Hz display cuts it further.</p> <p>Those numbers matter because the display is only the last step in a longer pipeline:</p> <ol> <li>The app decides what should change</li> <li>The operating system and graphics stack compose the frame</li> <li>The GPU renders it</li> <li>The display controller sends the result to the panel</li> <li>The panel updates its pixels</li> </ol> <p>If any stage misses the deadline, the user feels it. The symptom may look like lag, stutter, judder, or tearing, but the root problem is usually timing. A screen is a machine with recurring appointments, and modern interfaces feel good when the whole system shows up on time.</p> <h2 id="why-text-looks-sharp-and-gradients-look-hard">Why Text Looks Sharp and Gradients Look Hard<a class="zola-anchor" href="#why-text-looks-sharp-and-gradients-look-hard" aria-label="Anchor link for: why-text-looks-sharp-and-gradients-look-hard" style="visibility: hidden;"></a> </h2> <p>Screens are asked to do two very different jobs well.</p> <p>The first job is precision. Text, icons, and UI chrome need hard edges and stable alignment. That is where pixel density helps. Pack more pixels into the same physical area and the edges of letters become less jagged. Curves look less like stairs. Fine details survive zooming and resizing.</p> <p>The second job is subtlety. Photos, shadows, and gradients need smooth changes in light and color. That depends on how accurately the panel can represent brightness steps and color transitions. If the screen does a poor job here, skies band, skin tones look wrong, and dark scenes lose detail.</p> <p>This is why two displays with the same resolution can feel very different. Resolution answers "how many dots do I get?" It does not answer "how well can each dot behave?"</p> <h2 id="lcd-and-oled-solve-the-same-problem-differently">LCD and OLED Solve the Same Problem Differently<a class="zola-anchor" href="#lcd-and-oled-solve-the-same-problem-differently" aria-label="Anchor link for: lcd-and-oled-solve-the-same-problem-differently" style="visibility: hidden;"></a> </h2> <p>Most modern screens fall into two broad families, and the difference is easiest to understand in terms of where the light comes from.</p> <h3 id="lcd-control-the-light">LCD: control the light<a class="zola-anchor" href="#lcd-control-the-light" aria-label="Anchor link for: lcd-control-the-light" style="visibility: hidden;"></a> </h3> <p>An LCD panel usually has a backlight behind the screen. The liquid crystal layer does not create light itself. It acts more like a gate, shaping how much light passes through colored filters.</p> <p>This design has advantages:</p> <ul> <li>Mature manufacturing</li> <li>High brightness</li> <li>Good longevity in many use cases</li> </ul> <p>But it also has a limit. If the backlight is on, blocking it perfectly is difficult. That is why dark scenes on many LCDs can look gray instead of truly black.</p> <h3 id="oled-emit-the-light">OLED: emit the light<a class="zola-anchor" href="#oled-emit-the-light" aria-label="Anchor link for: oled-emit-the-light" style="visibility: hidden;"></a> </h3> <p>In an OLED panel, each pixel emits its own light. If a pixel needs to be black, it can turn off entirely.</p> <p>This changes the experience immediately:</p> <ul> <li>Blacks look deeper</li> <li>Contrast feels stronger</li> <li>Dark interfaces often look richer</li> </ul> <p>The trade-off is that self-emissive pixels bring their own engineering constraints around brightness behavior, lifespan, and image retention risk. There is no free lunch in display technology. There are only different compromises made visible.</p> <h2 id="color-is-a-negotiation-between-math-and-human-vision">Color Is a Negotiation Between Math and Human Vision<a class="zola-anchor" href="#color-is-a-negotiation-between-math-and-human-vision" aria-label="Anchor link for: color-is-a-negotiation-between-math-and-human-vision" style="visibility: hidden;"></a> </h2> <p>A display does not show "red" the way paint shows red. It approximates color through combinations of subpixel output that your eyes and brain interpret as a target shade.</p> <p>That means color on screens is always a negotiation:</p> <ul> <li>The content is encoded in a color space</li> <li>The operating system and GPU translate it</li> <li>The display tries to reproduce it</li> <li>Your eyes interpret the final result in context</li> </ul> <p>This is why the same image can look punchy on one screen and dull on another. Saturation, white point, gamma behavior, ambient light, and calibration all influence the result. Screens are not merely technical devices. They are interpretation devices.</p> <h2 id="brightness-is-about-more-than-visibility">Brightness Is About More Than Visibility<a class="zola-anchor" href="#brightness-is-about-more-than-visibility" aria-label="Anchor link for: brightness-is-about-more-than-visibility" style="visibility: hidden;"></a> </h2> <p>We often treat brightness as a single number, but it changes the whole feel of a screen.</p> <p>Higher brightness helps outdoors, but brightness also affects:</p> <ul> <li>perceived contrast</li> <li>highlight detail in HDR scenes</li> <li>eye comfort in dark rooms</li> <li>battery life on portable devices</li> </ul> <p>A bright display that handles reflections poorly can still feel hard to use. A dimmer display with better contrast can feel clearer indoors. In practice, the quality of a screen comes from balance, not from winning a single specification battle.</p> <h2 id="latency-is-the-hidden-feature-everyone-notices">Latency Is the Hidden Feature Everyone Notices<a class="zola-anchor" href="#latency-is-the-hidden-feature-everyone-notices" aria-label="Anchor link for: latency-is-the-hidden-feature-everyone-notices" style="visibility: hidden;"></a> </h2> <p>People rarely buy a phone or monitor because of "latency" in the abstract. But they absolutely notice it the moment it improves.</p> <p>Low latency is what makes a touchscreen feel attached to your finger. It is what makes a mouse feel precise. It is what makes a fast game feel fair instead of slippery.</p> <p>The important point is that screen latency is not just about panel refresh. It is the accumulated delay across sensing, rendering, transfer, and pixel response. You feel the full chain, not any one component in isolation.</p> <p>That is why premium devices often feel better even when their spec sheets look close to cheaper ones. The difference is usually not a single breakthrough. It is disciplined optimization across the pipeline.</p> <h2 id="the-best-screens-hide-their-complexity">The Best Screens Hide Their Complexity<a class="zola-anchor" href="#the-best-screens-hide-their-complexity" aria-label="Anchor link for: the-best-screens-hide-their-complexity" style="visibility: hidden;"></a> </h2> <p>The real achievement of a great display is not that it dazzles you with technology. It is that it disappears.</p> <p>When a screen is doing its job well, you do not think about subpixels, scanout timing, response curves, or brightness management. You think about the article you are reading, the person you are calling, the game you are playing, or the design you are editing.</p> <p>That invisibility is the point. A screen wins when it turns engineering into intuition.</p> <h2 id="final-thought">Final Thought<a class="zola-anchor" href="#final-thought" aria-label="Anchor link for: final-thought" style="visibility: hidden;"></a> </h2> <p>Every display is performing a demanding act of coordination: millions of colored elements, refreshed on a strict schedule, tuned for human perception, and expected to feel instantaneous. We ask a thin slab of glass and electronics to behave like paper, cinema, paint, and live motion all at once.</p> <p>Once you understand that, it becomes hard not to respect the screen in front of you. It is not just showing an image. It is rebuilding one, over and over again, fast enough that you mistake the reconstruction for reality.</p> 2026-01-23T00:00:00+00:00 2026-01-23T00:00:00+00:00 https://rajnandan.com/posts/largest-triangle-three-buckets-downsampling/ <h2 id="the-core-problem">The Core Problem<a class="zola-anchor" href="#the-core-problem" aria-label="Anchor link for: the-core-problem" style="visibility: hidden;"></a> </h2> <p>You have 10 million time-series data points: stock prices, server metrics, IoT sensor readings. Your charting library chokes past 50,000 points, the tab freezes, and users complain.</p> <p>The naive fix, taking every Nth point, produces jagged lines and drops the peaks and valleys where anomalies live.</p> <p>Largest Triangle Three Buckets (LTTB) preserves the visual shape of your data by maximizing the area of triangles formed between consecutive points. You get charts that keep their shape and render fast.</p> <p>There is a <a href="/demos/lttb/">live demo</a> of everything below: drag the threshold, switch between LTTB and every-Nth sampling, and watch which points survive.</p> <h2 id="what-is-lttb">What Is LTTB?<a class="zola-anchor" href="#what-is-lttb" aria-label="Anchor link for: what-is-lttb" style="visibility: hidden;"></a> </h2> <p>LTTB is a downsampling algorithm that reduces N points to M points (where M &lt;&lt; N) while maintaining visual accuracy. The name describes how it works:</p> <ul> <li><strong>Largest Triangle:</strong> It finds the point that forms the largest triangle area with neighboring points</li> <li><strong>Three Buckets:</strong> It divides data into buckets and looks at three buckets at a time</li> </ul> <p>Imagine drawing a mountain range: you pick the peaks, valleys, and slopes that capture its shape instead of tracing every grain of sand. LTTB does this mathematically.</p> <h3 id="the-mathematical-insight">The Mathematical Insight<a class="zola-anchor" href="#the-mathematical-insight" aria-label="Anchor link for: the-mathematical-insight" style="visibility: hidden;"></a> </h3> <p>The algorithm maximizes the area of triangles formed by consecutive selected points. A large triangle area marks a significant visual change, which is what your eye needs to read the data's shape.</p> <p>For three points forming a triangle, the area formula is:</p> <pre class="z-code"><code><span class="z-text z-plain">area = |x₁(y₂ - y₃) + x₂(y₃ - y₁) + x₃(y₁ - y₂)| / 2 </span></code></pre> <p>By selecting points that maximize these areas, we preserve visual features: sharp turns, peaks, valleys, and trend changes.</p> <h2 id="why-use-lttb">Why Use LTTB?<a class="zola-anchor" href="#why-use-lttb" aria-label="Anchor link for: why-use-lttb" style="visibility: hidden;"></a> </h2> <h3 id="1-visual-fidelity-over-statistical-precision">1. Visual Fidelity Over Statistical Precision<a class="zola-anchor" href="#1-visual-fidelity-over-statistical-precision" aria-label="Anchor link for: 1-visual-fidelity-over-statistical-precision" style="visibility: hidden;"></a> </h3> <p>LTTB optimizes for what you see, not statistical accuracy. A rendered chart needs the shape of the data, not each point. LTTB preserves:</p> <ul> <li><strong>Peaks and valleys:</strong> Extreme values stay visible</li> <li><strong>Trend changes:</strong> Slope transitions remain clear</li> <li><strong>Visual density:</strong> High-activity regions get more points</li> </ul> <h3 id="2-predictable-performance">2. Predictable Performance<a class="zola-anchor" href="#2-predictable-performance" aria-label="Anchor link for: 2-predictable-performance" style="visibility: hidden;"></a> </h3> <p>The algorithm is O(n): one pass through your data, no sorting, no extra data structures. For 1 million points downsampled to 1,000:</p> <ul> <li><strong>Time complexity:</strong> O(n), a linear scan</li> <li><strong>Space complexity:</strong> O(1) beyond the output buffer</li> <li><strong>Processing time:</strong> ~10-50ms on modern hardware</li> </ul> <h3 id="3-deterministic-results">3. Deterministic Results<a class="zola-anchor" href="#3-deterministic-results" aria-label="Anchor link for: 3-deterministic-results" style="visibility: hidden;"></a> </h3> <p>Unlike sampling with randomness, LTTB always produces the same output for the same input. This matters for:</p> <ul> <li><strong>Reproducibility:</strong> Charts look identical across page reloads</li> <li><strong>Debugging:</strong> Issues are consistent and trackable</li> <li><strong>Caching:</strong> You can cache downsampled results reliably</li> </ul> <h2 id="the-lttb-algorithm-step-by-step">The LTTB Algorithm: Step by Step<a class="zola-anchor" href="#the-lttb-algorithm-step-by-step" aria-label="Anchor link for: the-lttb-algorithm-step-by-step" style="visibility: hidden;"></a> </h2> <p><strong>Setup:</strong></p> <ol> <li>You have N source points</li> <li>You want M target points (M &lt; N)</li> <li>First and last points are always included (preserves range)</li> </ol> <p><strong>Process:</strong></p> <ol> <li>Divide remaining N-2 points into M-2 buckets</li> <li>Always include the first point</li> <li>For each bucket: <ul> <li>Look at the previously selected point (from bucket i-1)</li> <li>Look at the average point of the next bucket (bucket i+1)</li> <li>Select the point in current bucket (bucket i) that forms the largest triangle with these two points</li> </ul> </li> <li>Always include the last point</li> </ol> <p><strong>Visual representation:</strong></p> <pre class="z-code"><code><span class="z-text z-plain">Bucket 1 Bucket 2 Bucket 3 Bucket 4 </span><span class="z-text z-plain"> [...] [...] [...] [...] </span><span class="z-text z-plain"> ↓ ↓ ↓ ↓ </span><span class="z-text z-plain"> Point A Point B Point C Point D </span><span class="z-text z-plain"> </span><span class="z-text z-plain">For Bucket 2: </span><span class="z-text z-plain">- Previous selected: Point A (from Bucket 1) </span><span class="z-text z-plain">- Next bucket average: avg(Bucket 3 points) </span><span class="z-text z-plain">- Select point from Bucket 2 that makes largest triangle </span></code></pre> <h2 id="typescript-implementation">TypeScript Implementation<a class="zola-anchor" href="#typescript-implementation" aria-label="Anchor link for: typescript-implementation" style="visibility: hidden;"></a> </h2> <p>A production-ready implementation:</p> <pre data-lang="typescript" class="language-typescript z-code"><code class="language-typescript" data-lang="typescript"><span class="z-source z-ts"><span class="z-meta z-interface z-ts"><span class="z-storage z-type z-interface z-ts">interface</span> <span class="z-entity z-name z-type z-interface z-ts">Point</span> <span class="z-punctuation z-definition z-block z-ts">{</span> </span></span><span class="z-source z-ts"><span class="z-meta z-interface z-ts"><span class="z-meta z-field z-declaration z-ts"> <span class="z-meta z-definition z-property z-ts"><span class="z-variable z-object z-property z-ts">x</span></span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-support z-type z-primitive z-ts">number</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span><span class="z-source z-ts"><span class="z-meta z-interface z-ts"><span class="z-meta z-field z-declaration z-ts"> <span class="z-meta z-definition z-property z-ts"><span class="z-variable z-object z-property z-ts">y</span></span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-support z-type z-primitive z-ts">number</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span><span class="z-source z-ts"><span class="z-meta z-interface z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-comment z-block z-documentation z-ts"><span class="z-punctuation z-definition z-comment z-ts">/**</span> </span></span><span class="z-source z-ts"><span class="z-comment z-block z-documentation z-ts"> * Largest Triangle Three Buckets downsampling algorithm </span></span><span class="z-source z-ts"><span class="z-comment z-block z-documentation z-ts"> * </span></span><span class="z-source z-ts"><span class="z-comment z-block z-documentation z-ts"> * <span class="z-storage z-type z-class z-jsdoc"><span class="z-punctuation z-definition z-block z-tag z-jsdoc">@</span>param</span> <span class="z-variable z-other z-jsdoc">data</span> - Source data points (must be sorted by x) </span></span><span class="z-source z-ts"><span class="z-comment z-block z-documentation z-ts"> * <span class="z-storage z-type z-class z-jsdoc"><span class="z-punctuation z-definition z-block z-tag z-jsdoc">@</span>param</span> <span class="z-variable z-other z-jsdoc">threshold</span> - Target number of points in output </span></span><span class="z-source z-ts"><span class="z-comment z-block z-documentation z-ts"> * <span class="z-storage z-type z-class z-jsdoc"><span class="z-punctuation z-definition z-block z-tag z-jsdoc">@</span>returns</span> Downsampled points that preserve visual shape </span></span><span class="z-source z-ts"><span class="z-comment z-block z-documentation z-ts"> <span class="z-punctuation z-definition z-comment z-ts">*/</span></span> </span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-storage z-type z-function z-ts">function</span> <span class="z-meta z-definition z-function z-ts"><span class="z-entity z-name z-function z-ts">largestTriangleThreeBuckets</span></span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-parameters z-ts"> <span class="z-variable z-parameter z-ts">data</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">Point</span><span class="z-meta z-type z-tuple z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span></span><span class="z-punctuation z-separator z-parameter z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-parameters z-ts"> <span class="z-variable z-parameter z-ts">threshold</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-support z-type z-primitive z-ts">number</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-parameters z-ts"><span class="z-meta z-type z-annotation z-ts"></span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span><span class="z-meta z-return z-type z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">Point</span><span class="z-meta z-type z-tuple z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span> </span><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">dataLength</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-variable z-other z-object z-ts">data</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-ts">length</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-whitespace z-comment z-leading z-ts"> </span><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> If threshold is greater than or equal to data length, return all data</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">threshold</span> <span class="z-keyword z-operator z-relational z-ts">&gt;=</span> <span class="z-variable z-other z-readwrite z-ts">dataLength</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-variable z-other z-readwrite z-ts">data</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-whitespace z-comment z-leading z-ts"> </span><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> If threshold is very small, return edge points</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">threshold</span> <span class="z-keyword z-operator z-relational z-ts">&lt;=</span> <span class="z-constant z-numeric z-decimal z-ts">2</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span><span class="z-variable z-other z-readwrite z-ts">data</span><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-constant z-numeric z-decimal z-ts">0</span><span class="z-meta z-brace z-square z-ts">]</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-ts">data</span><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-variable z-other z-readwrite z-ts">dataLength</span> <span class="z-keyword z-operator z-arithmetic z-ts">-</span> <span class="z-constant z-numeric z-decimal z-ts">1</span><span class="z-meta z-brace z-square z-ts">]</span></span><span class="z-meta z-brace z-square z-ts">]</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">sampled</span></span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">Point</span><span class="z-meta z-type z-tuple z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span> </span></span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-whitespace z-comment z-leading z-ts"> </span><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Always include the first point</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">sampled</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-ts">push</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">data</span><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-constant z-numeric z-decimal z-ts">0</span><span class="z-meta z-brace z-square z-ts">]</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-whitespace z-comment z-leading z-ts"> </span><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Bucket size. Leave room for start and endpoints</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">bucketSize</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">dataLength</span> <span class="z-keyword z-operator z-arithmetic z-ts">-</span> <span class="z-constant z-numeric z-decimal z-ts">2</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-operator z-arithmetic z-ts">/</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">threshold</span> <span class="z-keyword z-operator z-arithmetic z-ts">-</span> <span class="z-constant z-numeric z-decimal z-ts">2</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-whitespace z-comment z-leading z-ts"> </span><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Index of currently selected point in previous bucket</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">let</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-readwrite z-ts">a</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-constant z-numeric z-decimal z-ts">0</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-loop z-ts">for</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">let</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-readwrite z-ts">i</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-constant z-numeric z-decimal z-ts">0</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> <span class="z-variable z-other z-readwrite z-ts">i</span> <span class="z-keyword z-operator z-relational z-ts">&lt;</span> <span class="z-variable z-other z-readwrite z-ts">threshold</span> <span class="z-keyword z-operator z-arithmetic z-ts">-</span> <span class="z-constant z-numeric z-decimal z-ts">2</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> <span class="z-variable z-other z-readwrite z-ts">i</span><span class="z-keyword z-operator z-increment z-ts">++</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-whitespace z-comment z-leading z-ts"> </span><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Calculate point average for next bucket (used as point C in triangle)</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">let</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-readwrite z-ts">avgX</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-constant z-numeric z-decimal z-ts">0</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">let</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-readwrite z-ts">avgY</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-constant z-numeric z-decimal z-ts">0</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">avgRangeStart</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-support z-constant z-math z-ts">Math</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-math z-ts">floor</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">i</span> <span class="z-keyword z-operator z-arithmetic z-ts">+</span> <span class="z-constant z-numeric z-decimal z-ts">1</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-operator z-arithmetic z-ts">*</span> <span class="z-variable z-other z-readwrite z-ts">bucketSize</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-operator z-arithmetic z-ts">+</span> <span class="z-constant z-numeric z-decimal z-ts">1</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">avgRangeEnd</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-support z-constant z-math z-ts">Math</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-math z-ts">min</span></span><span class="z-meta z-brace z-round z-ts">(</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-support z-constant z-math z-ts">Math</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-math z-ts">floor</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">i</span> <span class="z-keyword z-operator z-arithmetic z-ts">+</span> <span class="z-constant z-numeric z-decimal z-ts">2</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-operator z-arithmetic z-ts">*</span> <span class="z-variable z-other z-readwrite z-ts">bucketSize</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-operator z-arithmetic z-ts">+</span> <span class="z-constant z-numeric z-decimal z-ts">1</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-variable z-other z-readwrite z-ts">dataLength</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">avgRangeLength</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-variable z-other z-readwrite z-ts">avgRangeEnd</span> <span class="z-keyword z-operator z-arithmetic z-ts">-</span> <span class="z-variable z-other z-readwrite z-ts">avgRangeStart</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-loop z-ts">for</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">let</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-readwrite z-ts">j</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-variable z-other z-readwrite z-ts">avgRangeStart</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> <span class="z-variable z-other z-readwrite z-ts">j</span> <span class="z-keyword z-operator z-relational z-ts">&lt;</span> <span class="z-variable z-other z-readwrite z-ts">avgRangeEnd</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> <span class="z-variable z-other z-readwrite z-ts">j</span><span class="z-keyword z-operator z-increment z-ts">++</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-other z-readwrite z-ts">avgX</span> <span class="z-keyword z-operator z-assignment z-compound z-ts">+=</span> <span class="z-variable z-other z-readwrite z-ts">data</span><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-variable z-other z-readwrite z-ts">j</span><span class="z-meta z-brace z-square z-ts">]</span></span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-dom z-ts">x</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-other z-readwrite z-ts">avgY</span> <span class="z-keyword z-operator z-assignment z-compound z-ts">+=</span> <span class="z-variable z-other z-readwrite z-ts">data</span><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-variable z-other z-readwrite z-ts">j</span><span class="z-meta z-brace z-square z-ts">]</span></span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-dom z-ts">y</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-other z-readwrite z-ts">avgX</span> <span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-variable z-other z-readwrite z-ts">avgRangeLength</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-other z-readwrite z-ts">avgY</span> <span class="z-keyword z-operator z-arithmetic z-ts">/</span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-variable z-other z-readwrite z-ts">avgRangeLength</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-whitespace z-comment z-leading z-ts"> </span><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Get the range for this bucket</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">rangeStart</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-support z-constant z-math z-ts">Math</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-math z-ts">floor</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">i</span> <span class="z-keyword z-operator z-arithmetic z-ts">*</span> <span class="z-variable z-other z-readwrite z-ts">bucketSize</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-operator z-arithmetic z-ts">+</span> <span class="z-constant z-numeric z-decimal z-ts">1</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">rangeEnd</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-support z-constant z-math z-ts">Math</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-math z-ts">floor</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">i</span> <span class="z-keyword z-operator z-arithmetic z-ts">+</span> <span class="z-constant z-numeric z-decimal z-ts">1</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-operator z-arithmetic z-ts">*</span> <span class="z-variable z-other z-readwrite z-ts">bucketSize</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-operator z-arithmetic z-ts">+</span> <span class="z-constant z-numeric z-decimal z-ts">1</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-whitespace z-comment z-leading z-ts"> </span><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Point a is the previously selected point</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">pointA</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-variable z-other z-readwrite z-ts">data</span><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-variable z-other z-readwrite z-ts">a</span><span class="z-meta z-brace z-square z-ts">]</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">let</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-readwrite z-ts">maxArea</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-keyword z-operator z-arithmetic z-ts">-</span><span class="z-constant z-numeric z-decimal z-ts">1</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">let</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-readwrite z-ts">maxAreaPoint</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-constant z-numeric z-decimal z-ts">0</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-whitespace z-comment z-leading z-ts"> </span><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Find point in current bucket that forms largest triangle</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-loop z-ts">for</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">let</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-readwrite z-ts">j</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-variable z-other z-readwrite z-ts">rangeStart</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> <span class="z-variable z-other z-readwrite z-ts">j</span> <span class="z-keyword z-operator z-relational z-ts">&lt;</span> <span class="z-variable z-other z-readwrite z-ts">rangeEnd</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> <span class="z-variable z-other z-readwrite z-ts">j</span><span class="z-keyword z-operator z-increment z-ts">++</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-whitespace z-comment z-leading z-ts"> </span><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Calculate triangle area using the cross product formula</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-whitespace z-comment z-leading z-ts"> </span><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> area = |x₁(y₂ - y₃) + x₂(y₃ - y₁) + x₃(y₁ - y₂)| / 2</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">area</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-support z-constant z-math z-ts">Math</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-math z-ts">abs</span></span><span class="z-meta z-brace z-round z-ts">(</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-object z-ts">pointA</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-dom z-ts">x</span> <span class="z-keyword z-operator z-arithmetic z-ts">-</span> <span class="z-variable z-other z-readwrite z-ts">avgX</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-operator z-arithmetic z-ts">*</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">data</span><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-variable z-other z-readwrite z-ts">j</span><span class="z-meta z-brace z-square z-ts">]</span></span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-dom z-ts">y</span> <span class="z-keyword z-operator z-arithmetic z-ts">-</span> <span class="z-variable z-other z-object z-ts">pointA</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-dom z-ts">y</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-operator z-arithmetic z-ts">-</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-object z-ts">pointA</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-dom z-ts">x</span> <span class="z-keyword z-operator z-arithmetic z-ts">-</span> <span class="z-variable z-other z-readwrite z-ts">data</span><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-variable z-other z-readwrite z-ts">j</span><span class="z-meta z-brace z-square z-ts">]</span></span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-dom z-ts">x</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-operator z-arithmetic z-ts">*</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">avgY</span> <span class="z-keyword z-operator z-arithmetic z-ts">-</span> <span class="z-variable z-other z-object z-ts">pointA</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-dom z-ts">y</span><span class="z-meta z-brace z-round z-ts">)</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-operator z-arithmetic z-ts">*</span> <span class="z-constant z-numeric z-decimal z-ts">0<span class="z-meta z-delimiter z-decimal z-period z-ts">.</span>5</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">area</span> <span class="z-keyword z-operator z-relational z-ts">&gt;</span> <span class="z-variable z-other z-readwrite z-ts">maxArea</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-other z-readwrite z-ts">maxArea</span> <span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-variable z-other z-readwrite z-ts">area</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-other z-readwrite z-ts">maxAreaPoint</span> <span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-variable z-other z-readwrite z-ts">j</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">sampled</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-ts">push</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">data</span><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-variable z-other z-readwrite z-ts">maxAreaPoint</span><span class="z-meta z-brace z-square z-ts">]</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-other z-readwrite z-ts">a</span> <span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-variable z-other z-readwrite z-ts">maxAreaPoint</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> <span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> This point is the next &quot;point A&quot;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-whitespace z-comment z-leading z-ts"> </span><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Always include the last point</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">sampled</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-ts">push</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">data</span><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-variable z-other z-readwrite z-ts">dataLength</span> <span class="z-keyword z-operator z-arithmetic z-ts">-</span> <span class="z-constant z-numeric z-decimal z-ts">1</span><span class="z-meta z-brace z-square z-ts">]</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-variable z-other z-readwrite z-ts">sampled</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></code></pre> <h3 id="example-usage">Example Usage<a class="zola-anchor" href="#example-usage" aria-label="Anchor link for: example-usage" style="visibility: hidden;"></a> </h3> <pre data-lang="typescript" class="language-typescript z-code"><code class="language-typescript" data-lang="typescript"><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Generate sample time-series data: a sine wave with noise</span> </span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-storage z-type z-function z-ts">function</span> <span class="z-meta z-definition z-function z-ts"><span class="z-entity z-name z-function z-ts">generateTimeSeriesData</span></span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">count</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-support z-type z-primitive z-ts">number</span></span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span><span class="z-meta z-return z-type z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">Point</span><span class="z-meta z-type z-tuple z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span> </span><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">data</span></span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">Point</span><span class="z-meta z-type z-tuple z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span> </span></span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-loop z-ts">for</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">let</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-readwrite z-ts">i</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-constant z-numeric z-decimal z-ts">0</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> <span class="z-variable z-other z-readwrite z-ts">i</span> <span class="z-keyword z-operator z-relational z-ts">&lt;</span> <span class="z-variable z-other z-readwrite z-ts">count</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> <span class="z-variable z-other z-readwrite z-ts">i</span><span class="z-keyword z-operator z-increment z-ts">++</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">x</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-variable z-other z-readwrite z-ts">i</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">y</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-support z-constant z-math z-ts">Math</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-math z-ts">sin</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">i</span> <span class="z-keyword z-operator z-arithmetic z-ts">/</span> <span class="z-constant z-numeric z-decimal z-ts">100</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-operator z-arithmetic z-ts">*</span> <span class="z-constant z-numeric z-decimal z-ts">50</span> <span class="z-keyword z-operator z-arithmetic z-ts">+</span> <span class="z-meta z-function-call z-ts"><span class="z-support z-constant z-math z-ts">Math</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-math z-ts">random</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-operator z-arithmetic z-ts">*</span> <span class="z-constant z-numeric z-decimal z-ts">10</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">data</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-ts">push</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">x</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">y</span> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-variable z-other z-readwrite z-ts">data</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Original data: 10,000 points</span> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">originalData</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">generateTimeSeriesData</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-constant z-numeric z-decimal z-ts">10000</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-support z-class z-console z-ts">console</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-console z-ts">log</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-template z-ts"><span class="z-punctuation z-definition z-string z-template z-begin z-ts">`</span>Original: <span class="z-meta z-template z-expression z-ts"><span class="z-punctuation z-definition z-template-expression z-begin z-ts">${</span></span><span class="z-meta z-template z-expression z-ts"><span class="z-meta z-embedded z-line z-ts"><span class="z-variable z-other z-object z-ts">originalData</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-ts">length</span></span><span class="z-punctuation z-definition z-template-expression z-end z-ts">}</span></span> points<span class="z-punctuation z-definition z-string z-template z-end z-ts">`</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Downsample to 500 points</span> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">downsampled</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">largestTriangleThreeBuckets</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">originalData</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-constant z-numeric z-decimal z-ts">500</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-support z-class z-console z-ts">console</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-console z-ts">log</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-template z-ts"><span class="z-punctuation z-definition z-string z-template z-begin z-ts">`</span>Downsampled: <span class="z-meta z-template z-expression z-ts"><span class="z-punctuation z-definition z-template-expression z-begin z-ts">${</span></span><span class="z-meta z-template z-expression z-ts"><span class="z-meta z-embedded z-line z-ts"><span class="z-variable z-other z-object z-ts">downsampled</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-ts">length</span></span><span class="z-punctuation z-definition z-template-expression z-end z-ts">}</span></span> points<span class="z-punctuation z-definition z-string z-template z-end z-ts">`</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-support z-class z-console z-ts">console</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-console z-ts">log</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-template z-ts"><span class="z-punctuation z-definition z-string z-template z-begin z-ts">`</span>Reduction: <span class="z-meta z-template z-expression z-ts"><span class="z-punctuation z-definition z-template-expression z-begin z-ts">${</span></span><span class="z-meta z-template z-expression z-ts"><span class="z-meta z-embedded z-line z-ts"><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-constant z-numeric z-decimal z-ts">1</span> <span class="z-keyword z-operator z-arithmetic z-ts">-</span> <span class="z-variable z-other z-object z-ts">downsampled</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-ts">length</span> <span class="z-keyword z-operator z-arithmetic z-ts">/</span> <span class="z-variable z-other z-object z-ts">originalData</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-ts">length</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-operator z-arithmetic z-ts">*</span> <span class="z-constant z-numeric z-decimal z-ts">100</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-function-call z-ts"><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">toFixed</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-constant z-numeric z-decimal z-ts">1</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-definition z-template-expression z-end z-ts">}</span></span>%<span class="z-punctuation z-definition z-string z-template z-end z-ts">`</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Use in your charting library</span> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> chart.setData(downsampled); // Much faster rendering!</span> </span></code></pre> <h2 id="real-world-example-stock-price-visualization">Real-World Example: Stock Price Visualization<a class="zola-anchor" href="#real-world-example-stock-price-visualization" aria-label="Anchor link for: real-world-example-stock-price-visualization" style="visibility: hidden;"></a> </h2> <p>Say you're building a stock charting application. You fetch intraday data, one price per minute for a year:</p> <pre data-lang="typescript" class="language-typescript z-code"><code class="language-typescript" data-lang="typescript"><span class="z-source z-ts"><span class="z-meta z-interface z-ts"><span class="z-storage z-type z-interface z-ts">interface</span> <span class="z-entity z-name z-type z-interface z-ts">StockPrice</span> <span class="z-punctuation z-definition z-block z-ts">{</span> </span></span><span class="z-source z-ts"><span class="z-meta z-interface z-ts"><span class="z-meta z-field z-declaration z-ts"> <span class="z-meta z-definition z-property z-ts"><span class="z-variable z-object z-property z-ts">timestamp</span></span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-support z-type z-primitive z-ts">number</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> <span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Unix timestamp</span> </span></span><span class="z-source z-ts"><span class="z-meta z-interface z-ts"><span class="z-meta z-field z-declaration z-ts"> <span class="z-meta z-definition z-property z-ts"><span class="z-variable z-object z-property z-ts">price</span></span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-support z-type z-primitive z-ts">number</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span><span class="z-source z-ts"><span class="z-meta z-interface z-ts"><span class="z-meta z-field z-declaration z-ts"> <span class="z-meta z-definition z-property z-ts"><span class="z-variable z-object z-property z-ts">volume</span></span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-support z-type z-primitive z-ts">number</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span><span class="z-source z-ts"><span class="z-meta z-interface z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-storage z-type z-function z-ts">function</span> <span class="z-meta z-definition z-function z-ts"><span class="z-entity z-name z-function z-ts">downsampleStockData</span></span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-parameters z-ts"> <span class="z-variable z-parameter z-ts">prices</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">StockPrice</span><span class="z-meta z-type z-tuple z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span></span><span class="z-punctuation z-separator z-parameter z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-parameters z-ts"> <span class="z-variable z-parameter z-ts">targetPoints</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-support z-type z-primitive z-ts">number</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-parameters z-ts"><span class="z-meta z-type z-annotation z-ts"></span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span><span class="z-meta z-return z-type z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">StockPrice</span><span class="z-meta z-type z-tuple z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span> </span><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-whitespace z-comment z-leading z-ts"> </span><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Convert to Point format</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">points</span></span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">Point</span><span class="z-meta z-type z-tuple z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span> </span></span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">prices</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">map</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-arrow z-ts"><span class="z-variable z-parameter z-ts">p</span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> </span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">x</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-variable z-other z-object z-ts">p</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">timestamp</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">y</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-variable z-other z-object z-ts">p</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">price</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-whitespace z-comment z-leading z-ts"> </span><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Apply LTTB</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">downsampled</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">largestTriangleThreeBuckets</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">points</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-ts">targetPoints</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-whitespace z-comment z-leading z-ts"> </span><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Map back to original indices to preserve volume data</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">downsampledPrices</span></span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">StockPrice</span><span class="z-meta z-type z-tuple z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span> </span></span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">let</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-readwrite z-ts">priceIndex</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-constant z-numeric z-decimal z-ts">0</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-loop z-ts">for</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">point</span></span> </span></span><span class="z-keyword z-operator z-expression z-of z-ts">of</span> <span class="z-variable z-other z-readwrite z-ts">downsampled</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-whitespace z-comment z-leading z-ts"> </span><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Find original data point</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-loop z-ts">while</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">priceIndex</span> <span class="z-keyword z-operator z-relational z-ts">&lt;</span> <span class="z-variable z-other z-object z-ts">prices</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-ts">length</span> <span class="z-keyword z-operator z-logical z-ts">&amp;&amp;</span> <span class="z-variable z-other z-readwrite z-ts">prices</span><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-variable z-other z-readwrite z-ts">priceIndex</span><span class="z-meta z-brace z-square z-ts">]</span></span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">timestamp</span> <span class="z-keyword z-operator z-comparison z-ts">!==</span> <span class="z-variable z-other z-object z-ts">point</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-dom z-ts">x</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-other z-readwrite z-ts">priceIndex</span><span class="z-keyword z-operator z-increment z-ts">++</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">priceIndex</span> <span class="z-keyword z-operator z-relational z-ts">&lt;</span> <span class="z-variable z-other z-object z-ts">prices</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-ts">length</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">downsampledPrices</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-ts">push</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">prices</span><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-variable z-other z-readwrite z-ts">priceIndex</span><span class="z-meta z-brace z-square z-ts">]</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-variable z-other z-readwrite z-ts">downsampledPrices</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Usage: 1 year of minute data (525,600 points) down to 2,000</span> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">yearData</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">fetchStockPrices</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>AAPL<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>1y<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>1m<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> <span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> 525,600 points</span> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">chartData</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">downsampleStockData</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">yearData</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-constant z-numeric z-decimal z-ts">2000</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> <span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> 99.6% reduction</span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Chart renders instantly, all major price movements visible</span> </span></code></pre> <h2 id="where-to-use-lttb">Where to Use LTTB<a class="zola-anchor" href="#where-to-use-lttb" aria-label="Anchor link for: where-to-use-lttb" style="visibility: hidden;"></a> </h2> <h3 id="1-time-series-visualization">1. Time-Series Visualization<a class="zola-anchor" href="#1-time-series-visualization" aria-label="Anchor link for: 1-time-series-visualization" style="visibility: hidden;"></a> </h3> <p><strong>Perfect fit:</strong></p> <ul> <li>Financial charts (stocks, crypto, forex)</li> <li>Server metrics (CPU, memory, network)</li> <li>IoT sensor data (temperature, pressure, vibration)</li> <li>Analytics dashboards (user activity, sales trends)</li> </ul> <p><strong>Why it works:</strong> Time-series data is dense and continuous. LTTB preserves the temporal patterns you need to spot trends and anomalies.</p> <h3 id="2-real-time-monitoring-dashboards">2. Real-Time Monitoring Dashboards<a class="zola-anchor" href="#2-real-time-monitoring-dashboards" aria-label="Anchor link for: 2-real-time-monitoring-dashboards" style="visibility: hidden;"></a> </h3> <p>When streaming live metrics:</p> <pre data-lang="typescript" class="language-typescript z-code"><code class="language-typescript" data-lang="typescript"><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-storage z-type z-class z-ts">class</span> <span class="z-entity z-name z-type z-class z-ts">MetricsBuffer</span> <span class="z-punctuation z-definition z-block z-ts">{</span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"> <span class="z-storage z-modifier z-ts">private</span><span class="z-meta z-field z-declaration z-ts"> <span class="z-meta z-definition z-property z-ts"><span class="z-variable z-object z-property z-ts">buffer</span></span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">Point</span><span class="z-meta z-type z-tuple z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"> <span class="z-storage z-modifier z-ts">private</span><span class="z-meta z-field z-declaration z-ts"> <span class="z-meta z-definition z-property z-ts"><span class="z-variable z-object z-property z-ts">maxBufferSize</span></span> <span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-constant z-numeric z-decimal z-ts">100000</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"> <span class="z-storage z-modifier z-ts">private</span><span class="z-meta z-field z-declaration z-ts"> <span class="z-meta z-definition z-property z-ts"><span class="z-variable z-object z-property z-ts">displayThreshold</span></span> <span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-constant z-numeric z-decimal z-ts">1000</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"> <span class="z-meta z-definition z-method z-ts"><span class="z-entity z-name z-function z-ts">add</span></span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">point</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">Point</span></span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span><span class="z-meta z-return z-type z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-support z-type z-primitive z-ts">void</span> </span><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-object z-property z-ts">buffer</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-ts">push</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">point</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-whitespace z-comment z-leading z-ts"> </span><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Downsample when buffer gets large</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-object z-property z-ts">buffer</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-ts">length</span> <span class="z-keyword z-operator z-relational z-ts">&gt;</span> <span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">maxBufferSize</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">buffer</span> <span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">largestTriangleThreeBuckets</span></span><span class="z-meta z-brace z-round z-ts">(</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">buffer</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">maxBufferSize</span> <span class="z-keyword z-operator z-arithmetic z-ts">/</span> <span class="z-constant z-numeric z-decimal z-ts">2</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"> <span class="z-meta z-definition z-method z-ts"><span class="z-entity z-name z-function z-ts">getDisplayData</span></span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span><span class="z-meta z-return z-type z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">Point</span><span class="z-meta z-type z-tuple z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span> </span><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-object z-property z-ts">buffer</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-ts">length</span> <span class="z-keyword z-operator z-relational z-ts">&lt;=</span> <span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">displayThreshold</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">buffer</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">largestTriangleThreeBuckets</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">buffer</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">displayThreshold</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></code></pre> <h3 id="3-historical-data-analysis">3. Historical Data Analysis<a class="zola-anchor" href="#3-historical-data-analysis" aria-label="Anchor link for: 3-historical-data-analysis" style="visibility: hidden;"></a> </h3> <p>When users zoom out to see long time ranges:</p> <pre data-lang="typescript" class="language-typescript z-code"><code class="language-typescript" data-lang="typescript"><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-storage z-type z-function z-ts">function</span> <span class="z-meta z-definition z-function z-ts"><span class="z-entity z-name z-function z-ts">adaptiveDownsample</span></span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-parameters z-ts"> <span class="z-variable z-parameter z-ts">data</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">Point</span><span class="z-meta z-type z-tuple z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span></span><span class="z-punctuation z-separator z-parameter z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-parameters z-ts"> <span class="z-variable z-parameter z-ts">viewportWidth</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-support z-type z-primitive z-ts">number</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-parameters z-ts"><span class="z-meta z-type z-annotation z-ts"></span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span><span class="z-meta z-return z-type z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">Point</span><span class="z-meta z-type z-tuple z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span> </span><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-whitespace z-comment z-leading z-ts"> </span><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> One point per 2-3 pixels is optimal for line charts</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">targetPoints</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-support z-constant z-math z-ts">Math</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-math z-ts">floor</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">viewportWidth</span> <span class="z-keyword z-operator z-arithmetic z-ts">/</span> <span class="z-constant z-numeric z-decimal z-ts">2</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-object z-ts">data</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-ts">length</span> <span class="z-keyword z-operator z-relational z-ts">&lt;=</span> <span class="z-variable z-other z-readwrite z-ts">targetPoints</span> <span class="z-keyword z-operator z-arithmetic z-ts">*</span> <span class="z-constant z-numeric z-decimal z-ts">2</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-variable z-other z-readwrite z-ts">data</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">largestTriangleThreeBuckets</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">data</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-ts">targetPoints</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> When user zooms in/out, re-downsample for current view</span> </span><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">chart</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">on</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>zoom<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span><span class="z-meta z-arrow z-ts"> <span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">viewport</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">visible</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">getDataInViewport</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">allData</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-ts">viewport</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">downsampled</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">adaptiveDownsample</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">visible</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-object z-ts">viewport</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-dom z-ts">width</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">chart</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">update</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">downsampled</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></code></pre> <h3 id="4-data-export-transfer">4. Data Export/Transfer<a class="zola-anchor" href="#4-data-export-transfer" aria-label="Anchor link for: 4-data-export-transfer" style="visibility: hidden;"></a> </h3> <p>Reduce payload size for client applications:</p> <pre data-lang="typescript" class="language-typescript z-code"><code class="language-typescript" data-lang="typescript"><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> API endpoint that returns downsampled data</span> </span><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">app</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">get</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>/api/metrics/:id<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-arrow z-ts"><span class="z-storage z-modifier z-async z-ts">async</span> <span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">req</span><span class="z-punctuation z-separator z-parameter z-ts">,</span> <span class="z-variable z-parameter z-ts">res</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-object-binding-pattern-variable z-ts"><span class="z-punctuation z-definition z-binding-pattern z-object z-ts">{</span> <span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">points</span></span> <span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-constant z-numeric z-decimal z-ts">1000</span> <span class="z-punctuation z-definition z-binding-pattern z-object z-ts">}</span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-variable z-other z-object z-ts">req</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">query</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">fullData</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-keyword z-control z-flow z-ts">await</span> <span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">db</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">getMetrics</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-object z-ts">req</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-object z-property z-ts">params</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-dom z-ts">id</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-whitespace z-comment z-leading z-ts"> </span><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Downsample before sending</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">optimized</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">largestTriangleThreeBuckets</span></span><span class="z-meta z-brace z-round z-ts">(</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-variable z-other z-readwrite z-ts">fullData</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-support z-constant z-math z-ts">Math</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-math z-ts">min</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-function-call z-ts"><span class="z-support z-class z-builtin z-ts">Number</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">points</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-constant z-numeric z-decimal z-ts">5000</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Cap at 5k</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">res</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">json</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">original_count</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-variable z-other z-object z-ts">fullData</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-ts">length</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">downsampled_count</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-variable z-other z-object z-ts">optimized</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-ts">length</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">data</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-variable z-other z-readwrite z-ts">optimized</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></code></pre> <h2 id="where-not-to-use-lttb">Where NOT to Use LTTB<a class="zola-anchor" href="#where-not-to-use-lttb" aria-label="Anchor link for: where-not-to-use-lttb" style="visibility: hidden;"></a> </h2> <h3 id="1-statistical-analysis">1. Statistical Analysis<a class="zola-anchor" href="#1-statistical-analysis" aria-label="Anchor link for: 1-statistical-analysis" style="visibility: hidden;"></a> </h3> <p><strong>Don't use for:</strong></p> <ul> <li>Calculating averages, medians, or percentiles</li> <li>Standard deviation or variance</li> <li>Correlation analysis</li> <li>Any statistical computation</li> </ul> <p><strong>Why:</strong> LTTB optimizes for visual accuracy, not statistical properties. It biases toward extremes and changes, which skews statistics.</p> <pre data-lang="typescript" class="language-typescript z-code"><code class="language-typescript" data-lang="typescript"><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> BAD: Statistical analysis on downsampled data</span> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">downsampled</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">largestTriangleThreeBuckets</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">data</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-constant z-numeric z-decimal z-ts">1000</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">average</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">downsampled</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">reduce</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-arrow z-ts"><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">s</span><span class="z-punctuation z-separator z-parameter z-ts">,</span> <span class="z-variable z-parameter z-ts">p</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> </span><span class="z-variable z-other z-readwrite z-ts">s</span> <span class="z-keyword z-operator z-arithmetic z-ts">+</span> <span class="z-variable z-other z-object z-ts">p</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-dom z-ts">y</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-constant z-numeric z-decimal z-ts">0</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-operator z-arithmetic z-ts">/</span> <span class="z-variable z-other z-object z-ts">downsampled</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-ts">length</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> ❌ This average is WRONG - biased toward peaks/valleys</span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> GOOD: Calculate stats on full data</span> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">average</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">data</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">reduce</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-arrow z-ts"><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">s</span><span class="z-punctuation z-separator z-parameter z-ts">,</span> <span class="z-variable z-parameter z-ts">p</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> </span><span class="z-variable z-other z-readwrite z-ts">s</span> <span class="z-keyword z-operator z-arithmetic z-ts">+</span> <span class="z-variable z-other z-object z-ts">p</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-dom z-ts">y</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-constant z-numeric z-decimal z-ts">0</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-operator z-arithmetic z-ts">/</span> <span class="z-variable z-other z-object z-ts">data</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-ts">length</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> ✅ Correct statistical average</span> </span></code></pre> <h3 id="2-scientific-precision-requirements">2. Scientific Precision Requirements<a class="zola-anchor" href="#2-scientific-precision-requirements" aria-label="Anchor link for: 2-scientific-precision-requirements" style="visibility: hidden;"></a> </h3> <p><strong>Don't use for:</strong></p> <ul> <li>Medical device data where every reading matters</li> <li>Financial audit trails (compliance requires complete data)</li> <li>Scientific experiments requiring exact measurements</li> <li>Legal or regulatory data (must be complete and unaltered)</li> </ul> <p><strong>Reason:</strong> Downsampling discards information. If you need provable accuracy or regulatory compliance, you can't afford to lose any data points.</p> <h3 id="3-sparse-or-non-continuous-data">3. Sparse or Non-Continuous Data<a class="zola-anchor" href="#3-sparse-or-non-continuous-data" aria-label="Anchor link for: 3-sparse-or-non-continuous-data" style="visibility: hidden;"></a> </h3> <p><strong>Don't use for:</strong></p> <ul> <li>Event logs (sparse, discrete events)</li> <li>Transaction records (each record is unique)</li> <li>Categorical data (non-numeric or discrete categories)</li> <li>Already small datasets (&lt; 1000 points)</li> </ul> <p><strong>Example of bad fit:</strong></p> <pre data-lang="typescript" class="language-typescript z-code"><code class="language-typescript" data-lang="typescript"><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> BAD: Transaction log data</span> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">transactions</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">time</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-constant z-numeric z-decimal z-ts">100</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">type</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>purchase<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">amount</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-constant z-numeric z-decimal z-ts">50</span> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">time</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-constant z-numeric z-decimal z-ts">500</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">type</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>refund<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">amount</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-constant z-numeric z-decimal z-ts">20</span> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">time</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-constant z-numeric z-decimal z-ts">1000</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">type</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>purchase<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">amount</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-constant z-numeric z-decimal z-ts">100</span> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-brace z-square z-ts">]</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Each transaction is important - can&#39;t downsample without losing critical info</span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> BAD: Sparse sensor data with gaps</span> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">sensorData</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">time</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-constant z-numeric z-decimal z-ts">0</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">value</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-constant z-numeric z-decimal z-ts">10</span> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">time</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-constant z-numeric z-decimal z-ts">1000</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">value</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-constant z-numeric z-decimal z-ts">12</span> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> 1000 time units gap</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">time</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-constant z-numeric z-decimal z-ts">1001</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">value</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-constant z-numeric z-decimal z-ts">50</span> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span> <span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Sudden spike</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-brace z-square z-ts">]</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> The algorithm assumes continuous data - gaps break this assumption</span> </span></code></pre> <h3 id="4-when-you-need-exact-points">4. When You Need Exact Points<a class="zola-anchor" href="#4-when-you-need-exact-points" aria-label="Anchor link for: 4-when-you-need-exact-points" style="visibility: hidden;"></a> </h3> <p><strong>Don't use for:</strong></p> <ul> <li>Min/Max calculations (unless you verify endpoints)</li> <li>Exact zero-crossing detection</li> <li>Precise integration or area-under-curve</li> <li>Anomaly detection algorithms</li> </ul> <p>LTTB may miss the exact maximum if it falls inside a bucket but doesn't form a large triangle:</p> <pre data-lang="typescript" class="language-typescript z-code"><code class="language-typescript" data-lang="typescript"><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Potential issue: Missing exact peak</span> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">data</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">x</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-constant z-numeric z-decimal z-ts">0</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">y</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-constant z-numeric z-decimal z-ts">10</span> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">x</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-constant z-numeric z-decimal z-ts">1</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">y</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-constant z-numeric z-decimal z-ts">11</span> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">x</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-constant z-numeric z-decimal z-ts">2</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">y</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-constant z-numeric z-decimal z-ts">100</span> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Peak</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">x</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-constant z-numeric z-decimal z-ts">3</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">y</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-constant z-numeric z-decimal z-ts">11</span> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">x</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-constant z-numeric z-decimal z-ts">4</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">y</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-constant z-numeric z-decimal z-ts">10</span> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-brace z-square z-ts">]</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">downsampled</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">largestTriangleThreeBuckets</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">data</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-constant z-numeric z-decimal z-ts">3</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Might skip x=2 if it doesn&#39;t form largest triangle</span> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Use max aggregation instead for guaranteed peak capture</span> </span></code></pre> <h2 id="trade-offs-and-limitations">Trade-offs and Limitations<a class="zola-anchor" href="#trade-offs-and-limitations" aria-label="Anchor link for: trade-offs-and-limitations" style="visibility: hidden;"></a> </h2> <h3 id="memory-vs-visual-accuracy">Memory vs. Visual Accuracy<a class="zola-anchor" href="#memory-vs-visual-accuracy" aria-label="Anchor link for: memory-vs-visual-accuracy" style="visibility: hidden;"></a> </h3> <p>LTTB gives you a knob: the threshold parameter. Higher threshold = more points = better visual accuracy but slower rendering and more memory.</p> <p><strong>Sweet spots by use case:</strong></p> <ul> <li><strong>Live monitoring:</strong> 500-1,000 points (updates every second)</li> <li><strong>Historical analysis:</strong> 1,000-2,000 points (user-triggered)</li> <li><strong>Export/reporting:</strong> 2,000-5,000 points (one-time generation)</li> </ul> <h3 id="processing-cost">Processing Cost<a class="zola-anchor" href="#processing-cost" aria-label="Anchor link for: processing-cost" style="visibility: hidden;"></a> </h3> <p>The O(n) scan isn't free:</p> <pre data-lang="typescript" class="language-typescript z-code"><code class="language-typescript" data-lang="typescript"><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Benchmark on MacBook Pro M1</span> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">sizes</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span><span class="z-constant z-numeric z-decimal z-ts">10_000</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-constant z-numeric z-decimal z-ts">100_000</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-constant z-numeric z-decimal z-ts">1_000_000</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-constant z-numeric z-decimal z-ts">10_000_000</span><span class="z-meta z-brace z-square z-ts">]</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">sizes</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">forEach</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-arrow z-ts"><span class="z-variable z-parameter z-ts">size</span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">data</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">generateTimeSeriesData</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">size</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">start</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-support z-variable z-dom z-ts">performance</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">now</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">largestTriangleThreeBuckets</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">data</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-constant z-numeric z-decimal z-ts">1000</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">duration</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-support z-variable z-dom z-ts">performance</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">now</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-operator z-arithmetic z-ts">-</span> <span class="z-variable z-other z-readwrite z-ts">start</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-support z-class z-console z-ts">console</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-console z-ts">log</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-template z-ts"><span class="z-punctuation z-definition z-string z-template z-begin z-ts">`</span><span class="z-meta z-template z-expression z-ts"><span class="z-punctuation z-definition z-template-expression z-begin z-ts">${</span></span><span class="z-meta z-template z-expression z-ts"><span class="z-meta z-embedded z-line z-ts"><span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">size</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-ts">toLocaleString</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-definition z-template-expression z-end z-ts">}</span></span> points: <span class="z-meta z-template z-expression z-ts"><span class="z-punctuation z-definition z-template-expression z-begin z-ts">${</span></span><span class="z-meta z-template z-expression z-ts"><span class="z-meta z-embedded z-line z-ts"><span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">duration</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">toFixed</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-constant z-numeric z-decimal z-ts">2</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-definition z-template-expression z-end z-ts">}</span></span>ms<span class="z-punctuation z-definition z-string z-template z-end z-ts">`</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Results:</span> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> 10,000 points: 2ms</span> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> 100,000 points: 15ms</span> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> 1,000,000 points: 150ms</span> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> 10,000,000 points: 1,500ms</span> </span></code></pre> <p>For 10M points, 1.5 seconds is significant. Solutions:</p> <ul> <li><strong>Pre-compute:</strong> Downsample on the backend, cache results</li> <li><strong>Web Workers:</strong> Offload processing to background thread</li> <li><strong>Progressive loading:</strong> Start with coarse downsample, refine on idle</li> </ul> <pre data-lang="typescript" class="language-typescript z-code"><code class="language-typescript" data-lang="typescript"><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Web Worker approach</span> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> main.ts</span> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">worker</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-new z-expr z-ts"><span class="z-keyword z-operator z-new z-ts">new</span> <span class="z-entity z-name z-type z-ts">Worker</span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>lttb-worker.ts<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-meta z-brace z-round z-ts">)</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">worker</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">postMessage</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">data</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-variable z-other z-readwrite z-ts">largeDataset</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">threshold</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-constant z-numeric z-decimal z-ts">1000</span> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"><span class="z-variable z-other z-object z-ts">worker</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">onmessage</span> <span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-arrow z-ts"> <span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">e</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">chart</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">setData</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-object z-ts">e</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-dom z-ts">data</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">downsampled</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> lttb-worker.ts</span> </span><span class="z-source z-ts"><span class="z-variable z-other z-object z-ts">self</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">onmessage</span> <span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-arrow z-ts"> <span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">e</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-object-binding-pattern-variable z-ts"><span class="z-punctuation z-definition z-binding-pattern z-object z-ts">{</span> <span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">data</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">threshold</span></span> <span class="z-punctuation z-definition z-binding-pattern z-object z-ts">}</span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-variable z-other z-object z-ts">e</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-dom z-ts">data</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">downsampled</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">largestTriangleThreeBuckets</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">data</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-ts">threshold</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">self</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">postMessage</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-objectliteral z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-meta z-object z-member z-ts"><span class="z-variable z-other z-readwrite z-ts">downsampled</span> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></code></pre> <h3 id="loss-of-statistical-properties">Loss of Statistical Properties<a class="zola-anchor" href="#loss-of-statistical-properties" aria-label="Anchor link for: loss-of-statistical-properties" style="visibility: hidden;"></a> </h3> <p>The biggest limitation: downsampled data looks right on a chart and gives wrong numbers in calculations.</p> <table><thead><tr><th>Metric</th><th>Full Data</th><th>LTTB (1000pts)</th><th>Error</th></tr></thead><tbody> <tr><td>Mean</td><td>50.0</td><td>51.2</td><td>+2.4%</td></tr> <tr><td>Std Dev</td><td>10.5</td><td>12.8</td><td>+21.9%</td></tr> <tr><td>Min</td><td>25.0</td><td>25.0</td><td>0%</td></tr> <tr><td>Max</td><td>75.0</td><td>75.0</td><td>0%</td></tr> </tbody></table> <p><strong>Why:</strong> LTTB favors extremes. Standard deviation inflates because you're keeping more outliers relative to the mean.</p> <p><strong>Solution:</strong> Dual-path approach:</p> <pre data-lang="typescript" class="language-typescript z-code"><code class="language-typescript" data-lang="typescript"><span class="z-source z-ts"><span class="z-meta z-interface z-ts"><span class="z-storage z-type z-interface z-ts">interface</span> <span class="z-entity z-name z-type z-interface z-ts">DatasetView</span> <span class="z-punctuation z-definition z-block z-ts">{</span> </span></span><span class="z-source z-ts"><span class="z-meta z-interface z-ts"><span class="z-meta z-field z-declaration z-ts"> <span class="z-meta z-definition z-property z-ts"><span class="z-variable z-object z-property z-ts">visual</span></span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">Point</span><span class="z-meta z-type z-tuple z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> <span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> LTTB downsampled for charts</span> </span></span><span class="z-source z-ts"><span class="z-meta z-interface z-ts"><span class="z-meta z-field z-declaration z-ts"> <span class="z-meta z-definition z-property z-ts"><span class="z-variable z-object z-property z-ts">stats</span></span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">Point</span><span class="z-meta z-type z-tuple z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> <span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Full data or aggregated bins for calculations</span> </span></span><span class="z-source z-ts"><span class="z-meta z-interface z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-storage z-type z-function z-ts">function</span> <span class="z-meta z-definition z-function z-ts"><span class="z-entity z-name z-function z-ts">prepareDataset</span></span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">raw</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">Point</span><span class="z-meta z-type z-tuple z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span></span><span class="z-punctuation z-separator z-parameter z-ts">,</span> <span class="z-variable z-parameter z-ts">threshold</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-support z-type z-primitive z-ts">number</span></span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span><span class="z-meta z-return z-type z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">DatasetView</span> </span><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span><span class="z-meta z-objectliteral z-ts"> <span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">visual</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">largestTriangleThreeBuckets</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">raw</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-ts">threshold</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">stats</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-variable z-other z-readwrite z-ts">raw</span> <span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Keep full data for accurate calculations</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Use visual for rendering, stats for calculations</span> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">dataset</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">prepareDataset</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">rawData</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-constant z-numeric z-decimal z-ts">1000</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">chart</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">render</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-object z-ts">dataset</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">visual</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">average</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">calculateMean</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-object z-ts">dataset</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">stats</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></code></pre> <h2 id="advanced-patterns">Advanced Patterns<a class="zola-anchor" href="#advanced-patterns" aria-label="Anchor link for: advanced-patterns" style="visibility: hidden;"></a> </h2> <h3 id="multi-resolution-storage">Multi-Resolution Storage<a class="zola-anchor" href="#multi-resolution-storage" aria-label="Anchor link for: multi-resolution-storage" style="visibility: hidden;"></a> </h3> <p>Store multiple resolutions for different zoom levels:</p> <pre data-lang="typescript" class="language-typescript z-code"><code class="language-typescript" data-lang="typescript"><span class="z-source z-ts"><span class="z-meta z-interface z-ts"><span class="z-storage z-type z-interface z-ts">interface</span> <span class="z-entity z-name z-type z-interface z-ts">MultiResData</span> <span class="z-punctuation z-definition z-block z-ts">{</span> </span></span><span class="z-source z-ts"><span class="z-meta z-interface z-ts"><span class="z-meta z-field z-declaration z-ts"> <span class="z-meta z-definition z-property z-ts"><span class="z-variable z-object z-property z-ts">full</span></span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">Point</span><span class="z-meta z-type z-tuple z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span><span class="z-source z-ts"><span class="z-meta z-interface z-ts"><span class="z-meta z-field z-declaration z-ts"> <span class="z-meta z-definition z-property z-ts"><span class="z-variable z-object z-property z-ts">res_10k</span></span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">Point</span><span class="z-meta z-type z-tuple z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span><span class="z-source z-ts"><span class="z-meta z-interface z-ts"><span class="z-meta z-field z-declaration z-ts"> <span class="z-meta z-definition z-property z-ts"><span class="z-variable z-object z-property z-ts">res_1k</span></span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">Point</span><span class="z-meta z-type z-tuple z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span><span class="z-source z-ts"><span class="z-meta z-interface z-ts"><span class="z-meta z-field z-declaration z-ts"> <span class="z-meta z-definition z-property z-ts"><span class="z-variable z-object z-property z-ts">res_100</span></span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">Point</span><span class="z-meta z-type z-tuple z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span><span class="z-source z-ts"><span class="z-meta z-interface z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-storage z-type z-function z-ts">function</span> <span class="z-meta z-definition z-function z-ts"><span class="z-entity z-name z-function z-ts">createMultiResolution</span></span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">data</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">Point</span><span class="z-meta z-type z-tuple z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span></span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span><span class="z-meta z-return z-type z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">MultiResData</span> </span><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span><span class="z-meta z-objectliteral z-ts"> <span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">full</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-variable z-other z-readwrite z-ts">data</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">res_10k</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-variable z-other z-object z-ts">data</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-ts">length</span> <span class="z-keyword z-operator z-relational z-ts">&gt;</span> <span class="z-constant z-numeric z-decimal z-ts">10_000</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"> <span class="z-keyword z-operator z-ternary z-ts">?</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">largestTriangleThreeBuckets</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">data</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-constant z-numeric z-decimal z-ts">10_000</span><span class="z-meta z-brace z-round z-ts">)</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"> <span class="z-keyword z-operator z-ternary z-ts">:</span> <span class="z-variable z-other z-readwrite z-ts">data</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">res_1k</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">largestTriangleThreeBuckets</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">data</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-constant z-numeric z-decimal z-ts">1_000</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-objectliteral z-ts"> <span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts">res_100</span></span><span class="z-meta z-object z-member z-ts"><span class="z-meta z-object-literal z-key z-ts"><span class="z-punctuation z-separator z-key-value z-ts">:</span></span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">largestTriangleThreeBuckets</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">data</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-constant z-numeric z-decimal z-ts">100</span><span class="z-meta z-brace z-round z-ts">)</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-objectliteral z-ts"><span class="z-meta z-object z-member z-ts"> </span><span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-storage z-type z-function z-ts">function</span> <span class="z-meta z-definition z-function z-ts"><span class="z-entity z-name z-function z-ts">selectResolution</span></span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-parameters z-ts"> <span class="z-variable z-parameter z-ts">multi</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">MultiResData</span></span><span class="z-punctuation z-separator z-parameter z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-parameters z-ts"> <span class="z-variable z-parameter z-ts">visiblePoints</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-support z-type z-primitive z-ts">number</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-parameters z-ts"><span class="z-meta z-type z-annotation z-ts"></span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span><span class="z-meta z-return z-type z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">Point</span><span class="z-meta z-type z-tuple z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span> </span><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">visiblePoints</span> <span class="z-keyword z-operator z-relational z-ts">&gt;</span> <span class="z-constant z-numeric z-decimal z-ts">5_000</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-variable z-other z-object z-ts">multi</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">full</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">visiblePoints</span> <span class="z-keyword z-operator z-relational z-ts">&gt;</span> <span class="z-constant z-numeric z-decimal z-ts">500</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-variable z-other z-object z-ts">multi</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">res_10k</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">visiblePoints</span> <span class="z-keyword z-operator z-relational z-ts">&gt;</span> <span class="z-constant z-numeric z-decimal z-ts">50</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-variable z-other z-object z-ts">multi</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">res_1k</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-variable z-other z-object z-ts">multi</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">res_100</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></code></pre> <h3 id="streaming-incremental-updates">Streaming Incremental Updates<a class="zola-anchor" href="#streaming-incremental-updates" aria-label="Anchor link for: streaming-incremental-updates" style="visibility: hidden;"></a> </h3> <p>For real-time data, you can't reprocess everything on each new point:</p> <pre data-lang="typescript" class="language-typescript z-code"><code class="language-typescript" data-lang="typescript"><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-storage z-type z-class z-ts">class</span> <span class="z-entity z-name z-type z-class z-ts">StreamingLTTB</span> <span class="z-punctuation z-definition z-block z-ts">{</span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"> <span class="z-storage z-modifier z-ts">private</span><span class="z-meta z-field z-declaration z-ts"> <span class="z-meta z-definition z-property z-ts"><span class="z-variable z-object z-property z-ts">buffer</span></span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">Point</span><span class="z-meta z-type z-tuple z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"> <span class="z-storage z-modifier z-ts">private</span><span class="z-meta z-field z-declaration z-ts"> <span class="z-meta z-definition z-property z-ts"><span class="z-variable z-object z-property z-ts">downsampled</span></span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">Point</span><span class="z-meta z-type z-tuple z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"> <span class="z-storage z-modifier z-ts">private</span><span class="z-meta z-field z-declaration z-ts"> <span class="z-meta z-definition z-property z-ts"><span class="z-variable z-object z-property z-ts">windowSize</span></span> <span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-constant z-numeric z-decimal z-ts">10_000</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"> <span class="z-storage z-modifier z-ts">private</span><span class="z-meta z-field z-declaration z-ts"> <span class="z-meta z-definition z-property z-ts"><span class="z-variable z-object z-property z-ts">targetSize</span></span> <span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-constant z-numeric z-decimal z-ts">1_000</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"> <span class="z-meta z-definition z-method z-ts"><span class="z-entity z-name z-function z-ts">add</span></span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">point</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">Point</span></span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span><span class="z-meta z-return z-type z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-support z-type z-primitive z-ts">void</span> </span><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-object z-property z-ts">buffer</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-ts">push</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">point</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-whitespace z-comment z-leading z-ts"> </span><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Sliding window approach</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-object z-property z-ts">buffer</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-ts">length</span> <span class="z-keyword z-operator z-relational z-ts">&gt;</span> <span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">windowSize</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-object z-property z-ts">buffer</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-ts">shift</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> <span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Remove oldest</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">recompute</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> <span class="z-keyword z-control z-conditional z-ts">else</span> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-object z-property z-ts">buffer</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-ts">length</span> <span class="z-keyword z-operator z-arithmetic z-ts">%</span> <span class="z-constant z-numeric z-decimal z-ts">100</span> <span class="z-keyword z-operator z-comparison z-ts">===</span> <span class="z-constant z-numeric z-decimal z-ts">0</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-whitespace z-comment z-leading z-ts"> </span><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Recompute every 100 points</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">recompute</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"> <span class="z-meta z-method z-declaration z-ts"><span class="z-storage z-modifier z-ts">private</span> <span class="z-meta z-definition z-method z-ts"><span class="z-entity z-name z-function z-ts">recompute</span></span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span><span class="z-meta z-return z-type z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-support z-type z-primitive z-ts">void</span> </span><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">downsampled</span> <span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">largestTriangleThreeBuckets</span></span><span class="z-meta z-brace z-round z-ts">(</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">buffer</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">targetSize</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"> <span class="z-meta z-definition z-method z-ts"><span class="z-entity z-name z-function z-ts">getView</span></span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span><span class="z-meta z-return z-type z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">Point</span><span class="z-meta z-type z-tuple z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span> </span><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">downsampled</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></code></pre> <h2 id="comparison-with-other-downsampling-methods">Comparison with Other Downsampling Methods<a class="zola-anchor" href="#comparison-with-other-downsampling-methods" aria-label="Anchor link for: comparison-with-other-downsampling-methods" style="visibility: hidden;"></a> </h2> <table><thead><tr><th>Method</th><th>Pros</th><th>Cons</th><th>Use Case</th></tr></thead><tbody> <tr><td><strong>Every Nth</strong></td><td>Simple, O(1)</td><td>Misses peaks/valleys</td><td>Never (use LTTB instead)</td></tr> <tr><td><strong>Random Sampling</strong></td><td>Unbiased stats</td><td>Unpredictable visual</td><td>Statistical analysis</td></tr> <tr><td><strong>Min-Max</strong></td><td>Preserves extremes</td><td>Loses smooth curves</td><td>Range-bound displays</td></tr> <tr><td><strong>LTTB</strong></td><td>Visual fidelity, O(n)</td><td>Stats biased</td><td>Time-series charts</td></tr> <tr><td><strong>M4</strong></td><td>Very fast, extremes</td><td>Complex, requires binning</td><td>High-frequency data</td></tr> </tbody></table> <p><strong>When to choose LTTB:</strong></p> <ul> <li>You're rendering line charts</li> <li>Visual shape matters more than exact statistics</li> <li>Data is time-ordered and continuous</li> <li>You need consistent, reproducible results</li> </ul> <h2 id="production-checklist">Production Checklist<a class="zola-anchor" href="#production-checklist" aria-label="Anchor link for: production-checklist" style="visibility: hidden;"></a> </h2> <p>Before deploying LTTB in production:</p> <ul> <li><input disabled="" type="checkbox"/> Verify input data is sorted by x-axis (time)</li> <li><input disabled="" type="checkbox"/> Handle edge cases: empty arrays, single points, threshold &gt;= data length</li> <li><input disabled="" type="checkbox"/> Add input validation: non-null points, valid numbers</li> <li><input disabled="" type="checkbox"/> Consider pre-computing and caching downsampled views</li> <li><input disabled="" type="checkbox"/> Use Web Workers for large datasets (&gt; 100k points)</li> <li><input disabled="" type="checkbox"/> Keep full data for statistical calculations</li> <li><input disabled="" type="checkbox"/> Document that downsampled data is for visualization only</li> <li><input disabled="" type="checkbox"/> Add metrics to monitor processing time</li> <li><input disabled="" type="checkbox"/> Test with real production data shapes (spikes, flat lines, noise)</li> </ul> <h2 id="conclusion">Conclusion<a class="zola-anchor" href="#conclusion" aria-label="Anchor link for: conclusion" style="visibility: hidden;"></a> </h2> <p>LTTB is a visualization optimization, not a general-purpose compression algorithm. It trades statistical accuracy for visual fidelity, and when you're rendering charts that trade is worth making. Your users see the same patterns in 1,000 points that exist in 1,000,000, and the browser stays responsive.</p> <p>Use it when you draw lines on screens. Keep the full data for math. The whole implementation fits in about 80 lines of TypeScript, and you can see it running in the <a href="/demos/lttb/">live demo</a>.</p> 2026-01-21T00:00:00+00:00 2026-01-21T00:00:00+00:00 https://rajnandan.com/posts/go-frontend-architecture-2026/ <p>For the last decade, the standard answer to "How do I build a UI with my Go backend?" was "You don't." You wrote a REST or gRPC API in Go, and you handed the keys to a Node.js team building a React Single Page Application (SPA).</p> <p>Architecture is cyclical. In 2026, the pendulum has swung firmly back to the server. The complexity of managing distributed state between a client (React) and a server (Go) has become a burden that simple "API-first" designs ignore.</p> <p>This post analyzes the two dominant architectural patterns for Go engineers today: the <strong>"Pure Go" Component Stack</strong> (enabled by tools like Templ) versus the <strong>"Headless" Go Stack</strong> (Go as a dumb JSON pipe for Next.js).</p> <h2 id="the-problem-the-props-chasm">The Problem: The "Props" Chasm<a class="zola-anchor" href="#the-problem-the-props-chasm" aria-label="Anchor link for: the-problem-the-props-chasm" style="visibility: hidden;"></a> </h2> <p>The core friction in modern web development isn't rendering; it's data synchronization.</p> <p>When you split your stack into Go (backend) and TypeScript (frontend), you create a "Props Chasm." To render a simple user profile, you must:</p> <ol> <li>Define a <code>User</code> struct in Go.</li> <li>Serialize it to JSON.</li> <li>Transmit it over the wire (latency).</li> <li>Parse it in JavaScript.</li> <li>Validate it with Zod/Yup (runtime overhead).</li> <li>Define a matching TypeScript interface.</li> </ol> <p>If you change a field in Go, your frontend silently breaks or renders <code>undefined</code> until you redeploy. Tools like OpenAPI generate client code to mitigate this, but they are band-aids over a serialization wound.</p> <h2 id="approach-1-the-pure-go-stack-templ-htmx">Approach 1: The "Pure Go" Stack (Templ + HTMX)<a class="zola-anchor" href="#approach-1-the-pure-go-stack-templ-htmx" aria-label="Anchor link for: approach-1-the-pure-go-stack-templ-htmx" style="visibility: hidden;"></a> </h2> <p>The most significant shift in the Go ecosystem is the maturation of <strong>Templ</strong>.</p> <p>Templ is not a string-based template engine like <code>html/template</code>. It is a transpiler that converts Go-like HTML syntax into pure, efficient Go code. It brings the "Component" mental model - props, composition, type safety - to the backend.</p> <h3 id="the-component-model">The Component Model<a class="zola-anchor" href="#the-component-model" aria-label="Anchor link for: the-component-model" style="visibility: hidden;"></a> </h3> <p>In React, you write:</p> <pre data-lang="tsx" class="language-tsx z-code"><code class="language-tsx" data-lang="tsx"><span class="z-source z-tsx"><span class="z-comment z-line z-double-slash z-tsx"><span class="z-punctuation z-definition z-comment z-tsx">//</span></span><span class="z-comment z-line z-double-slash z-tsx"> Profile.tsx</span> </span><span class="z-source z-tsx"><span class="z-meta z-interface z-tsx"><span class="z-storage z-type z-interface z-tsx">interface</span> <span class="z-entity z-name z-type z-interface z-tsx">Props</span> <span class="z-punctuation z-definition z-block z-tsx">{</span> </span></span><span class="z-source z-tsx"><span class="z-meta z-interface z-tsx"><span class="z-meta z-field z-declaration z-tsx"> <span class="z-meta z-definition z-property z-tsx"><span class="z-variable z-object z-property z-tsx">user</span></span><span class="z-meta z-type z-annotation z-tsx"><span class="z-keyword z-operator z-type z-annotation z-tsx">:</span> <span class="z-entity z-name z-type z-tsx">User</span></span></span><span class="z-punctuation z-terminator z-statement z-tsx">;</span> </span></span><span class="z-source z-tsx"><span class="z-meta z-interface z-tsx"><span class="z-punctuation z-definition z-block z-tsx">}</span></span> </span><span class="z-source z-tsx"> </span><span class="z-source z-tsx"><span class="z-meta z-var z-expr z-tsx"><span class="z-keyword z-control z-export z-tsx">export</span> <span class="z-storage z-type z-tsx">const</span> <span class="z-meta z-var-single-variable z-expr z-tsx"><span class="z-meta z-definition z-variable z-tsx"><span class="z-variable z-other z-constant z-tsx"><span class="z-entity z-name z-function z-tsx">Profile</span></span></span> </span><span class="z-keyword z-operator z-assignment z-tsx">=</span><span class="z-meta z-arrow z-tsx"> <span class="z-meta z-parameters z-tsx"><span class="z-punctuation z-definition z-parameters z-begin z-tsx">(</span><span class="z-meta z-parameter z-object-binding-pattern z-tsx"><span class="z-punctuation z-definition z-binding-pattern z-object z-tsx">{</span> <span class="z-variable z-parameter z-tsx">user</span> <span class="z-punctuation z-definition z-binding-pattern z-object z-tsx">}</span></span><span class="z-meta z-type z-annotation z-tsx"><span class="z-keyword z-operator z-type z-annotation z-tsx">:</span> <span class="z-entity z-name z-type z-tsx">Props</span></span><span class="z-punctuation z-definition z-parameters z-end z-tsx">)</span></span> </span><span class="z-meta z-arrow z-tsx"><span class="z-storage z-type z-function z-arrow z-tsx">=&gt;</span> <span class="z-meta z-brace z-round z-tsx">(</span> </span></span></span><span class="z-source z-tsx"><span class="z-meta z-var z-expr z-tsx"><span class="z-meta z-arrow z-tsx"> <span class="z-meta z-tag z-tsx"><span class="z-punctuation z-definition z-tag z-begin z-tsx">&lt;</span><span class="z-entity z-name z-tag z-tsx">div</span><span class="z-meta z-tag z-attributes z-tsx"> <span class="z-entity z-other z-attribute-name z-tsx">className</span><span class="z-keyword z-operator z-assignment z-tsx">=</span><span class="z-string z-quoted z-double z-tsx"><span class="z-punctuation z-definition z-string z-begin z-tsx">&quot;</span>p-4 border<span class="z-punctuation z-definition z-string z-end z-tsx">&quot;</span></span></span><span class="z-punctuation z-definition z-tag z-end z-tsx">&gt;</span><span class="z-meta z-jsx z-children z-tsx"> </span></span></span></span></span><span class="z-source z-tsx"><span class="z-meta z-var z-expr z-tsx"><span class="z-meta z-arrow z-tsx"><span class="z-meta z-tag z-tsx"><span class="z-meta z-jsx z-children z-tsx"> <span class="z-meta z-tag z-without-attributes z-tsx"><span class="z-punctuation z-definition z-tag z-begin z-tsx">&lt;</span><span class="z-entity z-name z-tag z-tsx">h1</span><span class="z-punctuation z-definition z-tag z-end z-tsx">&gt;</span></span><span class="z-meta z-tag z-without-attributes z-tsx"><span class="z-meta z-jsx z-children z-tsx"><span class="z-punctuation z-section z-embedded z-begin z-tsx">{</span><span class="z-meta z-embedded z-expression z-tsx"><span class="z-variable z-other z-object z-tsx">user</span><span class="z-punctuation z-accessor z-tsx">.</span><span class="z-variable z-other z-property z-tsx">Name</span></span><span class="z-punctuation z-section z-embedded z-end z-tsx">}</span></span><span class="z-punctuation z-definition z-tag z-begin z-tsx">&lt;/</span><span class="z-entity z-name z-tag z-tsx">h1</span><span class="z-punctuation z-definition z-tag z-end z-tsx">&gt;</span></span> </span></span></span></span></span><span class="z-source z-tsx"><span class="z-meta z-var z-expr z-tsx"><span class="z-meta z-arrow z-tsx"><span class="z-meta z-tag z-tsx"><span class="z-meta z-jsx z-children z-tsx"> <span class="z-punctuation z-section z-embedded z-begin z-tsx">{</span><span class="z-meta z-embedded z-expression z-tsx"><span class="z-variable z-other z-object z-tsx">user</span><span class="z-punctuation z-accessor z-tsx">.</span><span class="z-variable z-other z-property z-tsx">IsAdmin</span> <span class="z-keyword z-operator z-logical z-tsx">&amp;&amp;</span> <span class="z-meta z-tag z-tsx"><span class="z-punctuation z-definition z-tag z-begin z-tsx">&lt;</span><span class="z-entity z-name z-tag z-tsx"><span class="z-support z-class z-component z-tsx">AdminBadge</span></span><span class="z-meta z-tag z-attributes z-tsx"> <span class="z-entity z-other z-attribute-name z-tsx">role</span><span class="z-keyword z-operator z-assignment z-tsx">=</span><span class="z-punctuation z-section z-embedded z-begin z-tsx">{</span><span class="z-meta z-embedded z-expression z-tsx"><span class="z-variable z-other z-object z-tsx">user</span><span class="z-punctuation z-accessor z-tsx">.</span><span class="z-variable z-other z-property z-tsx">Role</span></span><span class="z-punctuation z-section z-embedded z-end z-tsx">}</span> </span><span class="z-punctuation z-definition z-tag z-end z-tsx">/&gt;</span></span></span><span class="z-punctuation z-section z-embedded z-end z-tsx">}</span> </span></span></span></span></span><span class="z-source z-tsx"><span class="z-meta z-var z-expr z-tsx"><span class="z-meta z-arrow z-tsx"><span class="z-meta z-tag z-tsx"><span class="z-meta z-jsx z-children z-tsx"> </span><span class="z-punctuation z-definition z-tag z-begin z-tsx">&lt;/</span><span class="z-entity z-name z-tag z-tsx">div</span><span class="z-punctuation z-definition z-tag z-end z-tsx">&gt;</span></span> </span></span></span><span class="z-source z-tsx"><span class="z-meta z-var z-expr z-tsx"><span class="z-meta z-arrow z-tsx"><span class="z-meta z-brace z-round z-tsx">)</span></span></span><span class="z-punctuation z-terminator z-statement z-tsx">;</span> </span></code></pre> <p>In Templ (2026), you write:</p> <pre data-lang="go" class="language-go z-code"><code class="language-go" data-lang="go"><span class="z-source z-go"><span class="z-comment z-line z-go"><span class="z-punctuation z-definition z-comment z-go">//</span> profile.templ </span></span><span class="z-source z-go"><span class="z-keyword z-other z-package z-go">package</span> <span class="z-variable z-other z-go">components</span> </span><span class="z-source z-go"> </span><span class="z-source z-go"><span class="z-keyword z-other z-import z-go">import</span> <span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>github.com/myorg/types<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span> </span><span class="z-source z-go"> </span><span class="z-source z-go"><span class="z-variable z-other z-go">templ</span> <span class="z-variable z-function z-go">Profile</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">user</span> <span class="z-variable z-other z-go">types</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">User</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-operator z-go">&lt;</span><span class="z-variable z-other z-go">div</span> <span class="z-variable z-other z-go">class</span><span class="z-keyword z-operator z-assignment z-go">=</span><span class="z-string z-quoted z-double z-go"><span class="z-punctuation z-definition z-string z-begin z-go">&quot;</span>p-4 border<span class="z-punctuation z-definition z-string z-end z-go">&quot;</span></span><span class="z-keyword z-operator z-go">&gt;</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-operator z-go">&lt;</span><span class="z-variable z-other z-go">h1</span><span class="z-keyword z-operator z-go">&gt;</span><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> <span class="z-variable z-other z-go">user</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">Name</span> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span><span class="z-keyword z-operator z-go">&lt;</span><span class="z-keyword z-operator z-go">/</span><span class="z-variable z-other z-go">h1</span><span class="z-keyword z-operator z-go">&gt;</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-control z-go">if</span> <span class="z-variable z-other z-go">user</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">IsAdmin</span> <span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-begin z-go">{</span> </span></span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> @<span class="z-variable z-function z-go">AdminBadge</span><span class="z-punctuation z-section z-parens z-begin z-go">(</span><span class="z-variable z-other z-go">user</span><span class="z-punctuation z-accessor z-dot z-go">.</span><span class="z-variable z-other z-member z-go">Role</span><span class="z-punctuation z-section z-parens z-end z-go">)</span> </span></span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-meta z-block z-go"> <span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"> <span class="z-keyword z-operator z-go">&lt;</span><span class="z-keyword z-operator z-go">/</span><span class="z-variable z-other z-go">div</span><span class="z-keyword z-operator z-go">&gt;</span> </span></span><span class="z-source z-go"><span class="z-meta z-block z-go"><span class="z-punctuation z-section z-braces z-end z-go">}</span></span> </span></code></pre> <h3 id="the-architectural-win-zero-cost-abstractions">The Architectural Win: Zero-Cost Abstractions<a class="zola-anchor" href="#the-architectural-win-zero-cost-abstractions" aria-label="Anchor link for: the-architectural-win-zero-cost-abstractions" style="visibility: hidden;"></a> </h3> <p>Because Templ compiles to standard Go functions, the "component tree" is just a function call tree.</p> <ul> <li><strong>Performance:</strong> There is no Virtual DOM. No reconciliation. It is just <code>io.Writer</code> streaming bytes.</li> <li><strong>Type Safety:</strong> If you remove <code>Role</code> from the <code>User</code> struct in Go, your UI code fails to compile. <strong>The feedback loop is immediate.</strong></li> <li><strong>Composition:</strong> You can pass components as arguments to other components, allowing for layouts and higher-order components exactly like React <code>children</code>.</li> </ul> <p>When paired with <strong>HTMX</strong> for interactivity (handling form submissions, partial page replacements), you eliminate the need for a separate Node.js build pipeline entirely.</p> <h2 id="approach-2-the-headless-go-stack-next-js-remix">Approach 2: The "Headless" Go Stack (Next.js / Remix)<a class="zola-anchor" href="#approach-2-the-headless-go-stack-next-js-remix" aria-label="Anchor link for: approach-2-the-headless-go-stack-next-js-remix" style="visibility: hidden;"></a> </h2> <p>The alternative is accepting that Go is terrible at UI state. If your application requires rich client-side interactivity - dragging nodes on a canvas, complex data grids, offline-first behavior - server-side rendering in Go hits a wall. Templ can render the <em>initial</em> HTML fast, but it cannot manage complex browser state efficiently.</p> <p>In this model, Go retreats to the "Backend for Frontend" (BFF) role.</p> <ul> <li><strong>Next.js</strong> handles the Rendering / Routing / State.</li> <li><strong>Go</strong> handles the Business Logic / Database / Queues.</li> </ul> <h3 id="the-serialization-tax">The Serialization Tax<a class="zola-anchor" href="#the-serialization-tax" aria-label="Anchor link for: the-serialization-tax" style="visibility: hidden;"></a> </h3> <p>The trade-off here is operational complexity. You are now running two distinct distributed systems. You need observability for both. You need to align deployments (so the API doesn't drift from the frontend). You are paying the serialization tax on every request.</p> <h2 id="the-verdict-how-to-choose-in-2026">The Verdict: How to Choose in 2026<a class="zola-anchor" href="#the-verdict-how-to-choose-in-2026" aria-label="Anchor link for: the-verdict-how-to-choose-in-2026" style="visibility: hidden;"></a> </h2> <p>As a Staff Engineer, avoid "defaulting" to React just because it's popular. Default to the architecture that minimizes moving parts.</p> <h3 id="choose-strict-go-templ-if">Choose strict Go + Templ if:<a class="zola-anchor" href="#choose-strict-go-templ-if" aria-label="Anchor link for: choose-strict-go-templ-if" style="visibility: hidden;"></a> </h3> <ol> <li><strong>Your app is "Content-Heavy" or "CRUD-Heavy":</strong> Dashboards, admin panels, e-commerce listings, blogs.</li> <li><strong>Team Composition:</strong> Your team is predominantly backend/systems engineers.</li> <li><strong>Performance Constraint:</strong> You need "Time to First Byte" (TTFB) under 50ms. Go streaming HTML is unbeatable here.</li> </ol> <h3 id="choose-go-react-next-js-if">Choose Go + React/Next.js if:<a class="zola-anchor" href="#choose-go-react-next-js-if" aria-label="Anchor link for: choose-go-react-next-js-if" style="visibility: hidden;"></a> </h3> <ol> <li><strong>Your app is "State-Heavy":</strong> collaborative editors (Figma-like), complex interactive visualizations, heavy use of browser APIs.</li> <li><strong>Ecosystem Reliance:</strong> You strictly need libraries that only exist in React (e.g., highly specific calendar widgets or map integrations).</li> <li><strong>Hiring:</strong> You plan to hire specialized frontend engineers who do not want to touch Go code.</li> </ol> <h2 id="conclusion">Conclusion<a class="zola-anchor" href="#conclusion" aria-label="Anchor link for: conclusion" style="visibility: hidden;"></a> </h2> <p>The era of "Go templating is messy strings" is over. With Templ, Go has a first-class component story that rivals Svelte or React for developer experience, with vastly superior performance characteristics.</p> <p>Before you spin up a <code>node_modules</code> folder, ask yourself: Do I need a distributed system, or do I just need to render some HTML?</p> 2026-01-21T00:00:00+00:00 2026-01-21T00:00:00+00:00 https://rajnandan.com/posts/state-of-agentic-commerce-2026/ <p>As of January 2026, the hypothetical concept of "Agentic Commerce" - software agents autonomously executing financial transactions - has transitioned from research papers to production infrastructure. The announcements at NRF (National Retail Federation) 2026 have solidified the battle lines for the next decade of digital retail.</p> <p>This post analyzes the current state of the ecosystem, focusing on the emerging "Protocol Wars" between major tech incumbents, and synthesizes strategic insights from recent consultancy reports.</p> <h2 id="the-infrastructure-divergence-acp-vs-ucp">The Infrastructure Divergence: ACP vs. UCP<a class="zola-anchor" href="#the-infrastructure-divergence-acp-vs-ucp" aria-label="Anchor link for: the-infrastructure-divergence-acp-vs-ucp" style="visibility: hidden;"></a> </h2> <p>The most significant development of early 2026 is the bifurcation of standards. We are witnessing a classic standards war, reminiscent of Betamax vs. VHS, but applied to the negotiation layer of AI commerce.</p> <h3 id="1-stripe-openai-the-agentic-commerce-protocol-acp">1. Stripe &amp; OpenAI: The Agentic Commerce Protocol (ACP)<a class="zola-anchor" href="#1-stripe-openai-the-agentic-commerce-protocol-acp" aria-label="Anchor link for: 1-stripe-openai-the-agentic-commerce-protocol-acp" style="visibility: hidden;"></a> </h3> <p><strong>Philosophy: Payment-Backwards</strong></p> <p>Stripe, in collaboration with OpenAI, has doubled down on the <strong>Agentic Commerce Protocol (ACP)</strong>. Their thesis is built on the friction of finality. Agents can browse easily, but they stumble at the checkout.</p> <ul> <li><strong>The Stack:</strong> ACP focuses on a standardized "handshake" for payment authorization. It creates a secure tunnel between an LLM (running on OpenAI's infrastructure) and a merchant's payment gateway (Stripe).</li> <li><strong>Key Feature - Agentic Commerce Suite:</strong> This newly announced suite allows retailers to "broadcast" their liquidity to agents. It treats AI models as a new distribution channel, distinct from web and mobile.</li> <li><strong>Adoption:</strong> The integration with ChatGPT's "Instant Checkout" is the flagship implementation, aiming to capture high-intent users who treat the chatbot as a search engine.</li> </ul> <h3 id="2-google-shopify-the-universal-commerce-protocol-ucp">2. Google &amp; Shopify: The Universal Commerce Protocol (UCP)<a class="zola-anchor" href="#2-google-shopify-the-universal-commerce-protocol-ucp" aria-label="Anchor link for: 2-google-shopify-the-universal-commerce-protocol-ucp" style="visibility: hidden;"></a> </h3> <p><strong>Philosophy: Search-Forwards</strong></p> <p>Announced at NRF 2026, the <strong>Universal Commerce Protocol (UCP)</strong> is a counter-move by an alliance of Google, Shopify, and major retailers like Walmart and Wayfair.</p> <ul> <li><strong>The Stack:</strong> UCP is broader than payments; it is a <em>negotiation</em> protocol. It defines how an agent discovers catalogue capabilities ("Do you have this in blue?" "Can you deliver by Tuesday?") before a transaction is even attempted.</li> <li><strong>The "Embedded Checkout Protocol" (ECP):</strong> A sub-component of UCP, this allows the transaction to happen directly within the discovery surface (e.g., Google's "AI Mode" in Search or Gemini).</li> <li><strong>Differentiation:</strong> Unlike ACP, which feels like a pipe between a brain and a wallet, UCP feels like a structured language for digital bartering. It is designed to handle the complexity of retail logic (inventory variances, shipping rules) upstream from the payment.</li> </ul> <h3 id="3-the-wildcard-x402-the-revival-of-http-402">3. The Wildcard: x402 &amp; The Revival of HTTP 402<a class="zola-anchor" href="#3-the-wildcard-x402-the-revival-of-http-402" aria-label="Anchor link for: 3-the-wildcard-x402-the-revival-of-http-402" style="visibility: hidden;"></a> </h3> <p><strong>Philosophy: The Native Web Standard</strong></p> <p>While Stripe and Google battle for the high-value retail cart, a third faction led by Cloudflare, Coinbase, and the open-source community is reviving the long-dormant <strong>HTTP 402 "Payment Required"</strong> status code.</p> <ul> <li><strong>The "x402" Standard:</strong> This is the protocol for the <em>agent economy</em> itself. It allows an API or web resource to block an agent with a <code>402</code> error, which includes a header requesting a micropayment (often 0.1¢ to 5¢).</li> <li><strong>Usage:</strong> Unlike ACP/UCP which sell <em>goods</em> (shoes, tickets), x402 sells <em>access</em> and <em>compute</em>. It is becoming the standard for agent-to-agent services - paying for clean data, premium inference, or specialized API lookups.</li> <li><strong>The Split:</strong> We are seeing a divergence: ACP/UCP for <em>Consumer Commerce</em> (Macro-payments &gt;$5) and x402 for <em>Machine Commerce</em> (Micro-payments &lt;$0.10).</li> </ul> <h2 id="strategic-insights-the-zero-billion-dollar-market">Strategic Insights: The "Zero Billion Dollar" Market<a class="zola-anchor" href="#strategic-insights-the-zero-billion-dollar-market" aria-label="Anchor link for: strategic-insights-the-zero-billion-dollar-market" style="visibility: hidden;"></a> </h2> <p>While the infrastructure is being laid, the actual market volume remains a subject of intense debate among analysts.</p> <h3 id="the-bull-case-mckinsey-the-merchandising-transformation">The Bull Case: McKinsey &amp; The Merchandising Transformation<a class="zola-anchor" href="#the-bull-case-mckinsey-the-merchandising-transformation" aria-label="Anchor link for: the-bull-case-mckinsey-the-merchandising-transformation" style="visibility: hidden;"></a> </h3> <p>In their 2026 analysis, McKinsey suggests that the primary value unlock isn't just in the transaction, but in the <em>operational</em> layer. They highlight that 65% of organizations are now using GenAI regularly, but the frontier is <strong>"Agentic Merchandising"</strong>.</p> <ul> <li><strong>Supply Chain Optimization:</strong> Agents don't just buy; they predict. McKinsey notes that retailers are using agentic AI to dynamically adjust pricing and COGS (Cost of Goods Sold) management in real-time response to agentic crawler demand.</li> <li><strong>The "Robotic Consumer":</strong> Brands are beginning to optimize their digital shelves not for human eyes (images, emotional copy) but for agent parsing (structured data, UCP compliance).</li> </ul> <h3 id="the-bear-case-the-assisted-reality">The Bear Case: The "Assisted" Reality<a class="zola-anchor" href="#the-bear-case-the-assisted-reality" aria-label="Anchor link for: the-bear-case-the-assisted-reality" style="visibility: hidden;"></a> </h3> <p>Contrasting the hype, analysts like Dan Rosen argue that "Agentic Commerce" as a fully autonomous market (where the human is out of the loop) is effectively a "$0 Billion Market" today.</p> <ul> <li><strong>Behavioral Friction:</strong> Consumers are using AI agents for <em>discovery</em> and <em>research</em>, but trust barriers prevent them from delegating the final "buy" button click for high-value items.</li> <li>** The "Assisted" Pivot:** The immediate opportunity is not autonomous buying, but <strong>Agent-Assisted Commerce</strong>. The protocols (ACP/UCP) are valuable because they structure the data for the <em>research</em> phase, even if a human eventually finalizes the purchase.</li> </ul> <h2 id="the-emerging-stack-2026">The Emerging Stack (2026)<a class="zola-anchor" href="#the-emerging-stack-2026" aria-label="Anchor link for: the-emerging-stack-2026" style="visibility: hidden;"></a> </h2> <p>To build for this ecosystem, developers must understand the four layers of the new stack:</p> <ol> <li><strong>The Brain (LLMs):</strong> GPT-5, Gemini Ultra.</li> <li><strong>The Context Layer (MCP):</strong> Anthropic’s <strong>Model Context Protocol</strong> remains the standard for agents to read data (pricing, specs) without hallucinations.</li> <li><strong>The Negotiation Layer (UCP/ACP):</strong> The new battleground. The choice between UCP and ACP will likely depend on your existing platform allegiance (Shopify vs. Custom/Stripe).</li> <li><strong>The Payment Layer (AP2):</strong> Google’s <strong>Agent Payments Protocol (AP2)</strong> is trying to abstract the payment method, making the agent "wallet-agnostic."</li> </ol> <h2 id="the-implementation-frontier-geographies-platforms">The Implementation Frontier: Geographies &amp; Platforms<a class="zola-anchor" href="#the-implementation-frontier-geographies-platforms" aria-label="Anchor link for: the-implementation-frontier-geographies-platforms" style="visibility: hidden;"></a> </h2> <p>As the stack matures, the implementation challenges are becoming strictly regional and platform-specific.</p> <h3 id="1-the-payment-rail-divide-tokenized-cards-vs-upi">1. The Payment Rail Divide: Tokenized Cards vs. UPI<a class="zola-anchor" href="#1-the-payment-rail-divide-tokenized-cards-vs-upi" aria-label="Anchor link for: 1-the-payment-rail-divide-tokenized-cards-vs-upi" style="visibility: hidden;"></a> </h3> <p>The biggest friction for global agentic commerce is the "Auth Layer."</p> <ul> <li><strong>US/West (The "Silent" Model):</strong> The US model relies on <strong>Tokenized Cards</strong> (Network Tokens). This is ideal for agents because a user can pre-authorize an agent (e.g., "Spend up to $50/month on coffee") and the agent can execute "silent" transactions without further user intervetion.</li> <li><strong>India/East (The "Interactive" Model):</strong> In India, <strong>UPI (Unified Payments Interface)</strong> dominates. While low-cost, standard UPI is high-friction for agents because it requires a PIN for every transaction. <ul> <li><em>The Workaround:</em> The rapid adoption of <strong>UPI Autopay</strong> and <strong>Delegated Mandates</strong> (e.g., UPI Lite for agents) is the bridge. However, this creates a fundamentally different UX: The Indian agent is "Assisted" (Ask for PIN) by default, whereas the US agent is "Autonomous" by default.</li> </ul> </li> </ul> <h3 id="2-strategic-geographies-ondc-vs-the-private-cloud">2. Strategic Geographies: ONDC vs. The Private Cloud<a class="zola-anchor" href="#2-strategic-geographies-ondc-vs-the-private-cloud" aria-label="Anchor link for: 2-strategic-geographies-ondc-vs-the-private-cloud" style="visibility: hidden;"></a> </h3> <p>Interestingly, while the US leads in <em>Model</em> capability (OpenAI/Google), India leads in <em>Network</em> readiness.</p> <ul> <li><strong>The private "Walled Gardens" (US):</strong> In the US, an agent needs UCP to talk to Shopify, ACP to talk to Stripe, and Amazon's proprietary API to talk to Amazon. It is fragmented.</li> <li><strong>The Public Standard (India's ONDC):</strong> The <strong>Beckn Protocol</strong> (the backbone of ONDC) effectively makes the <em>entire</em> Indian retail ecosystem "agent-ready" out of the box. An agent doesn't need to learn 50 APIs; it just needs to speak Beckn/ONDC to discover catalogs from groceries to ride-hailing.</li> </ul> <h3 id="3-the-death-of-the-app-interface">3. The Death of the "App" Interface<a class="zola-anchor" href="#3-the-death-of-the-app-interface" aria-label="Anchor link for: 3-the-death-of-the-app-interface" style="visibility: hidden;"></a> </h3> <p>What happens to the mobile app in an agentic world?</p> <ul> <li><strong>Shopify &amp; Headless:</strong> Shopify's push for <strong>"Agentic Storefronts"</strong> acknowledges that the website is no longer the primary destination. "Headless commerce" has shifted from a developer preference to a survival strategy. If your store data isn't exposed via API (JSON) for an agent to read, you are invisible.</li> <li><strong>Native Apps as "Dumb Pipes":</strong> Native mobile apps (iOS/Android) are the hardest for agents to navigate. We are seeing a shift where apps are becoming merely the "fulfillment layer" (tracking delivery), while the "discovery and transaction" layer moves entirely to the Agent's interface. Deep-linking relies on the agent "handing off" the user, but the goal of 2026 is to <em>maintain</em> the user in the conversation.</li> </ul> <h2 id="conclusion">Conclusion<a class="zola-anchor" href="#conclusion" aria-label="Anchor link for: conclusion" style="visibility: hidden;"></a> </h2> <p>2026 is the year of infrastructure. We have moved beyond the "demo" phase where an agent haphazardly navigates a web browser using vision. We now have dedicated pipes (ACP/UCP) for machine-to-machine commerce.</p> <p>For brands, the message is clear: You can no longer just optimize for SEO (Human Search). You must now optimize for <strong>AEO (Agent Environment Optimization)</strong>. If your catalog isn't readable by UCP or ACP, you are effectively invisible to the wealthiest consumers of the next decade - the AI agents.</p> 2026-01-20T00:00:00+00:00 2026-01-20T00:00:00+00:00 https://rajnandan.com/posts/apache-spark-flink-airflow-comparison/ <p>In the modern data engineering landscape, choosing the right tool for your data processing needs can be overwhelming. Apache Spark, Apache Flink, and Apache Airflow are three powerful frameworks that often come up in discussions, but they serve different purposes. In this comprehensive guide, we'll compare these three tools and provide real-world examples to help you understand when to use each one.</p> <h2 id="understanding-the-core-difference">Understanding the Core Difference<a class="zola-anchor" href="#understanding-the-core-difference" aria-label="Anchor link for: understanding-the-core-difference" style="visibility: hidden;"></a> </h2> <p>Before diving into the details, it's crucial to understand that these tools serve different primary purposes:</p> <ul> <li><strong>Apache Spark</strong>: A unified analytics engine for large-scale data processing (both batch and streaming)</li> <li><strong>Apache Flink</strong>: A stream processing framework with true real-time capabilities</li> <li><strong>Apache Airflow</strong>: A workflow orchestration platform for scheduling and monitoring data pipelines</li> </ul> <p>Think of it this way: Spark and Flink are the <strong>workers</strong> that process your data, while Airflow is the <strong>manager</strong> that schedules and coordinates when and how these workers operate.</p> <h2 id="apache-spark-the-unified-analytics-engine">Apache Spark: The Unified Analytics Engine<a class="zola-anchor" href="#apache-spark-the-unified-analytics-engine" aria-label="Anchor link for: apache-spark-the-unified-analytics-engine" style="visibility: hidden;"></a> </h2> <h3 id="what-is-apache-spark">What is Apache Spark?<a class="zola-anchor" href="#what-is-apache-spark" aria-label="Anchor link for: what-is-apache-spark" style="visibility: hidden;"></a> </h3> <p>Apache Spark is a distributed computing framework designed for large-scale data processing. It provides high-level APIs in Java, Scala, Python, and R, and supports batch processing, SQL queries, streaming data, machine learning, and graph processing.</p> <h3 id="key-features">Key Features<a class="zola-anchor" href="#key-features" aria-label="Anchor link for: key-features" style="visibility: hidden;"></a> </h3> <ul> <li><strong>In-memory processing</strong>: Processes data in RAM for faster computation</li> <li><strong>Lazy evaluation</strong>: Optimizes execution plans before running</li> <li><strong>RDD, DataFrame, and Dataset APIs</strong>: Multiple abstraction levels for different use cases</li> <li><strong>Spark Streaming</strong>: Micro-batch processing for near real-time analytics</li> <li><strong>MLlib</strong>: Built-in machine learning library</li> <li><strong>Spark SQL</strong>: SQL interface for structured data processing</li> </ul> <h3 id="real-world-example-e-commerce-user-behavior-analysis">Real-World Example: E-commerce User Behavior Analysis<a class="zola-anchor" href="#real-world-example-e-commerce-user-behavior-analysis" aria-label="Anchor link for: real-world-example-e-commerce-user-behavior-analysis" style="visibility: hidden;"></a> </h3> <p>Let's build a real-world example analyzing e-commerce user behavior to generate product recommendations.</p> <p><strong>Scenario</strong>: You have clickstream data from your e-commerce website stored in S3/HDFS, and you want to:</p> <ol> <li>Clean and transform the data</li> <li>Identify user purchase patterns</li> <li>Generate product recommendations</li> <li>Store results in a data warehouse</li> </ol> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-from z-python">from</span></span><span class="z-meta z-statement z-import z-python"><span class="z-meta z-import-source z-python"> <span class="z-meta z-import-path z-python"><span class="z-meta z-import-name z-python">pyspark</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">sql</span></span> <span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span></span></span></span><span class="z-meta z-statement z-import z-python"></span><span class="z-meta z-statement z-import z-python"> <span class="z-meta z-generic-name z-python">SparkSession</span></span> </span><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-from z-python">from</span></span><span class="z-meta z-statement z-import z-python"><span class="z-meta z-import-source z-python"> <span class="z-meta z-import-path z-python"><span class="z-meta z-import-name z-python">pyspark</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">sql</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">functions</span></span> <span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span></span></span></span><span class="z-meta z-statement z-import z-python"></span><span class="z-meta z-statement z-import z-python"> <span class="z-meta z-generic-name z-python">col</span><span class="z-punctuation z-separator z-import-list z-python">,</span> <span class="z-meta z-generic-name z-python">count</span><span class="z-punctuation z-separator z-import-list z-python">,</span> <span class="z-meta z-generic-name z-python">desc</span><span class="z-punctuation z-separator z-import-list z-python">,</span> <span class="z-meta z-generic-name z-python">collect_list</span><span class="z-punctuation z-separator z-import-list z-python">,</span> <span class="z-meta z-generic-name z-python">explode</span><span class="z-punctuation z-separator z-import-list z-python">,</span> <span class="z-meta z-generic-name z-python">lit</span><span class="z-punctuation z-separator z-import-list z-python">,</span> <span class="z-meta z-generic-name z-python">rank</span><span class="z-punctuation z-separator z-import-list z-python">,</span> <span class="z-support z-function z-builtin z-python">sum</span><span class="z-punctuation z-separator z-import-list z-python">,</span> <span class="z-meta z-generic-name z-python">avg</span><span class="z-punctuation z-separator z-import-list z-python">,</span> <span class="z-meta z-generic-name z-python">when</span><span class="z-punctuation z-separator z-import-list z-python">,</span> <span class="z-meta z-generic-name z-python">countDistinct</span></span> </span><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-from z-python">from</span></span><span class="z-meta z-statement z-import z-python"><span class="z-meta z-import-source z-python"> <span class="z-meta z-import-path z-python"><span class="z-meta z-import-name z-python">pyspark</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">sql</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">window</span></span> <span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span></span></span></span><span class="z-meta z-statement z-import z-python"></span><span class="z-meta z-statement z-import z-python"> <span class="z-meta z-generic-name z-python">Window</span></span> </span><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-from z-python">from</span></span><span class="z-meta z-statement z-import z-python"><span class="z-meta z-import-source z-python"> <span class="z-meta z-import-path z-python"><span class="z-meta z-import-name z-python">pyspark</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">ml</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">fpm</span></span> <span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span></span></span></span><span class="z-meta z-statement z-import z-python"></span><span class="z-meta z-statement z-import z-python"> <span class="z-meta z-generic-name z-python">FPGrowth</span></span> </span><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-from z-python">from</span></span><span class="z-meta z-statement z-import z-python"><span class="z-meta z-import-source z-python"> <span class="z-meta z-import-path z-python"><span class="z-meta z-import-name z-python">datetime</span></span> <span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span></span></span></span><span class="z-meta z-statement z-import z-python"></span><span class="z-meta z-statement z-import z-python"> <span class="z-meta z-generic-name z-python">datetime</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Initialize Spark Session </span></span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">spark</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">SparkSession</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">builder</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">appName</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">EcommerceUserBehaviorAnalysis<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">config</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">spark.sql.adaptive.enabled<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">true<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">config</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">spark.sql.adaptive.coalescePartitions.enabled<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">true<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">getOrCreate</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Read clickstream data from S3 </span></span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">clickstream_df</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">spark</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">read</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">option</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">header<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">true<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">option</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">inferSchema<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">true<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">parquet</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">s3a://your-bucket/clickstream-data/<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Sample data structure: </span></span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> user_id, session_id, event_type, product_id, category, timestamp, price </span></span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> 1. Data Cleaning and Transformation </span></span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">cleaned_df</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">clickstream_df</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">filter</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">col</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">event_type<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">isin</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-sequence z-list z-python"><span class="z-punctuation z-section z-sequence z-begin z-python">[</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">view<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">add_to_cart<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">purchase<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-sequence z-end z-python">]</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">filter</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">col</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">product_id<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">isNotNull</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">withColumn</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">date<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">col</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">timestamp<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">cast</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">date<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> 2. User Purchase Patterns Analysis </span></span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">user_purchases</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">cleaned_df</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">filter</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">col</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">event_type<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-keyword z-operator z-comparison z-python">==</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">purchase<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">groupBy</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">user_id<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">session_id<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">agg</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">collect_list</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">product_id<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">alias</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">products<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">count</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">product_id<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">alias</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">item_count<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">sum</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">price<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">alias</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">total_amount<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> 3. Find Popular Product Combinations using FP-Growth Algorithm </span></span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">fp_growth</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">FPGrowth</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-variable z-parameter z-python">itemsCol</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">products<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-variable z-parameter z-python">minSupport</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-constant z-numeric z-float z-decimal z-python">0<span class="z-punctuation z-separator z-decimal z-python">.</span>01</span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-variable z-parameter z-python">minConfidence</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-constant z-numeric z-float z-decimal z-python">0<span class="z-punctuation z-separator z-decimal z-python">.</span>3</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">model</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">fp_growth</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">fit</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">user_purchases</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Get frequent itemsets (products bought together) </span></span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">frequent_itemsets</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">model</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">freqItemsets</span></span> </span><span class="z-source z-python"><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">frequent_itemsets</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">orderBy</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">desc</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">freq<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">show</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-constant z-numeric z-integer z-decimal z-python">20</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Get association rules (if user buys A, they might buy B) </span></span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">association_rules</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">model</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">associationRules</span></span> </span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">recommendations_df</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">association_rules</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">withColumn</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">confidence_pct<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-group z-python"><span class="z-punctuation z-section z-group z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">col</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">confidence<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-keyword z-operator z-arithmetic z-python">*</span> <span class="z-constant z-numeric z-integer z-decimal z-python">100</span><span class="z-punctuation z-section z-group z-end z-python">)</span></span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">cast</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">int<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">filter</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">col</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">confidence<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-keyword z-operator z-comparison z-python">&gt;</span> <span class="z-constant z-numeric z-float z-decimal z-python">0<span class="z-punctuation z-separator z-decimal z-python">.</span>5</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">orderBy</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">desc</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">confidence<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> 4. Calculate Product Popularity by Category </span></span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">product_popularity</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">cleaned_df</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">filter</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">col</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">event_type<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-keyword z-operator z-comparison z-python">==</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">purchase<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">groupBy</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">category<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">product_id<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">agg</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">count</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">*<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">alias</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">purchase_count<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">avg</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">price<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">alias</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">avg_price<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Add popularity rank within each category </span></span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">window_spec</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">Window</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">partitionBy</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">category<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">orderBy</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">desc</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">purchase_count<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">product_popularity_ranked</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">product_popularity</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">withColumn</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">rank<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">rank</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">over</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">window_spec</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">filter</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">col</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">rank<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-keyword z-operator z-comparison z-python">&lt;=</span> <span class="z-constant z-numeric z-integer z-decimal z-python">10</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> 5. User Segmentation based on behavior </span></span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">user_segments</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">cleaned_df</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">groupBy</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">user_id<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">agg</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">count</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">when</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">col</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">event_type<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-keyword z-operator z-comparison z-python">==</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">view<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-constant z-numeric z-integer z-decimal z-python">1</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">alias</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">views<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">count</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">when</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">col</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">event_type<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-keyword z-operator z-comparison z-python">==</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">add_to_cart<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-constant z-numeric z-integer z-decimal z-python">1</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">alias</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">cart_adds<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">count</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">when</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">col</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">event_type<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-keyword z-operator z-comparison z-python">==</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">purchase<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-constant z-numeric z-integer z-decimal z-python">1</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">alias</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">purchases<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">sum</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">when</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">col</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">event_type<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-keyword z-operator z-comparison z-python">==</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">purchase<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">col</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">price<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">alias</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">total_spent<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">withColumn</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">conversion_rate<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-group z-python"><span class="z-punctuation z-section z-group z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">col</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">purchases<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-keyword z-operator z-arithmetic z-python">/</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">col</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">views<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-keyword z-operator z-arithmetic z-python">*</span> <span class="z-constant z-numeric z-integer z-decimal z-python">100</span><span class="z-punctuation z-section z-group z-end z-python">)</span></span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">cast</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">decimal(5,2)<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">withColumn</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">segment<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">when</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">col</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">total_spent<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-keyword z-operator z-comparison z-python">&gt;</span> <span class="z-constant z-numeric z-integer z-decimal z-python">1000</span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">premium<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">when</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">col</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">total_spent<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-keyword z-operator z-comparison z-python">&gt;</span> <span class="z-constant z-numeric z-integer z-decimal z-python">500</span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">regular<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">otherwise</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">casual<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> 6. Save results to data warehouse (e.g., Redshift, Snowflake) </span></span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">recommendations_df</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">write</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">mode</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">overwrite<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">format</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">jdbc<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">option</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">url<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">jdbc:postgresql://your-redshift-cluster:5439/database<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">option</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">dbtable<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">product_recommendations<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">option</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">user<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">username<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">option</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">password<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">password<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">save</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">product_popularity_ranked</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">write</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">mode</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">overwrite<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">format</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">jdbc<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">option</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">url<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">jdbc:postgresql://your-redshift-cluster:5439/database<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">option</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">dbtable<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">popular_products<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">option</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">user<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">username<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">option</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">password<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">password<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">save</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">user_segments</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">write</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">mode</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">overwrite<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">format</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">jdbc<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">option</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">url<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">jdbc:postgresql://your-redshift-cluster:5439/database<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">option</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">dbtable<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">user_segments<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">option</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">user<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">username<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">option</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">password<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">password<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">save</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> 7. Generate daily report </span></span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">daily_stats</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">cleaned_df</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">groupBy</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">date<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">event_type<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">agg</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">count</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">*<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">alias</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">event_count<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">countDistinct</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">user_id<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">alias</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">unique_users<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">sum</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">when</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">col</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">event_type<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-keyword z-operator z-comparison z-python">==</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">purchase<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">col</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">price<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">alias</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">revenue<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">daily_stats</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-generic-name z-python">write</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">mode</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">append<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">partitionBy</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">date<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-punctuation z-separator z-continuation z-line z-python">\</span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">parquet</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">s3a://your-bucket/daily-stats/<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">print</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">Analysis completed successfully!<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">spark</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">stop</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span></code></pre> <p><strong>Why Spark for this use case?</strong></p> <ul> <li>Handles large volumes of historical clickstream data efficiently</li> <li>Built-in machine learning library (FPGrowth) for pattern discovery</li> <li>Can process data from multiple sources (S3, HDFS, databases)</li> <li>Optimized for batch processing with in-memory computation</li> </ul> <h2 id="apache-flink-true-stream-processing">Apache Flink: True Stream Processing<a class="zola-anchor" href="#apache-flink-true-stream-processing" aria-label="Anchor link for: apache-flink-true-stream-processing" style="visibility: hidden;"></a> </h2> <h3 id="what-is-apache-flink">What is Apache Flink?<a class="zola-anchor" href="#what-is-apache-flink" aria-label="Anchor link for: what-is-apache-flink" style="visibility: hidden;"></a> </h3> <p>Apache Flink is a distributed stream processing framework that excels at processing unbounded data streams with low latency and high throughput. Unlike Spark's micro-batch approach, Flink provides true stream processing with event time semantics.</p> <h3 id="key-features-1">Key Features<a class="zola-anchor" href="#key-features-1" aria-label="Anchor link for: key-features-1" style="visibility: hidden;"></a> </h3> <ul> <li><strong>True streaming</strong>: Processes events one at a time, not in micro-batches</li> <li><strong>Event time processing</strong>: Handles out-of-order events correctly</li> <li><strong>Stateful computations</strong>: Maintains state across events</li> <li><strong>Exactly-once semantics</strong>: Guarantees no data loss or duplication</li> <li><strong>Low latency</strong>: Millisecond-level latency for real-time applications</li> <li><strong>Complex Event Processing (CEP)</strong>: Pattern detection in streams</li> </ul> <h3 id="real-world-example-real-time-fraud-detection-system">Real-World Example: Real-Time Fraud Detection System<a class="zola-anchor" href="#real-world-example-real-time-fraud-detection-system" aria-label="Anchor link for: real-world-example-real-time-fraud-detection-system" style="visibility: hidden;"></a> </h3> <p>Let's build a real-time fraud detection system for credit card transactions.</p> <p><strong>Scenario</strong>: Process credit card transactions in real-time to detect potential fraud patterns:</p> <ol> <li>Track transaction velocity per user</li> <li>Detect unusual spending patterns</li> <li>Flag transactions from new locations</li> <li>Identify suspicious transaction sequences</li> <li>Alert in real-time when fraud is detected</li> </ol> <pre data-lang="java" class="language-java z-code"><code class="language-java" data-lang="java"><span class="z-source z-java"><span class="z-meta z-import z-java"><span class="z-keyword z-control z-import z-java">import</span> <span class="z-meta z-path z-java"><span class="z-support z-type z-package z-java">org</span></span><span class="z-meta z-path z-java"><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">apache</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">flink</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">api</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">common</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">eventtime</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-class z-import z-java">WatermarkStrategy</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span><span class="z-source z-java"><span class="z-meta z-import z-java"><span class="z-keyword z-control z-import z-java">import</span> <span class="z-meta z-path z-java"><span class="z-support z-type z-package z-java">org</span></span><span class="z-meta z-path z-java"><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">apache</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">flink</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">api</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">common</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">functions</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-class z-import z-java">RichMapFunction</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span><span class="z-source z-java"><span class="z-meta z-import z-java"><span class="z-keyword z-control z-import z-java">import</span> <span class="z-meta z-path z-java"><span class="z-support z-type z-package z-java">org</span></span><span class="z-meta z-path z-java"><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">apache</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">flink</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">api</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">common</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">state</span><span class="z-punctuation z-accessor z-dot z-java">.</span></span><span class="z-meta z-path z-java"><span class="z-keyword z-operator z-wildcard z-asterisk z-java">*</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span><span class="z-source z-java"><span class="z-meta z-import z-java"><span class="z-keyword z-control z-import z-java">import</span> <span class="z-meta z-path z-java"><span class="z-support z-type z-package z-java">org</span></span><span class="z-meta z-path z-java"><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">apache</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">flink</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">api</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">common</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">typeinfo</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-class z-import z-java">TypeInformation</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span><span class="z-source z-java"><span class="z-meta z-import z-java"><span class="z-keyword z-control z-import z-java">import</span> <span class="z-meta z-path z-java"><span class="z-support z-type z-package z-java">org</span></span><span class="z-meta z-path z-java"><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">apache</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">flink</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">api</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">java</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">tuple</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-class z-import z-java">Tuple2</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span><span class="z-source z-java"><span class="z-meta z-import z-java"><span class="z-keyword z-control z-import z-java">import</span> <span class="z-meta z-path z-java"><span class="z-support z-type z-package z-java">org</span></span><span class="z-meta z-path z-java"><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">apache</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">flink</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">configuration</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-class z-import z-java">Configuration</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span><span class="z-source z-java"><span class="z-meta z-import z-java"><span class="z-keyword z-control z-import z-java">import</span> <span class="z-meta z-path z-java"><span class="z-support z-type z-package z-java">org</span></span><span class="z-meta z-path z-java"><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">apache</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">flink</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">streaming</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">api</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-class z-import z-java">CheckpointingMode</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span><span class="z-source z-java"><span class="z-meta z-import z-java"><span class="z-keyword z-control z-import z-java">import</span> <span class="z-meta z-path z-java"><span class="z-support z-type z-package z-java">org</span></span><span class="z-meta z-path z-java"><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">apache</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">flink</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">streaming</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">api</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">datastream</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-class z-import z-java">DataStream</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span><span class="z-source z-java"><span class="z-meta z-import z-java"><span class="z-keyword z-control z-import z-java">import</span> <span class="z-meta z-path z-java"><span class="z-support z-type z-package z-java">org</span></span><span class="z-meta z-path z-java"><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">apache</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">flink</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">streaming</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">api</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">environment</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-class z-import z-java">StreamExecutionEnvironment</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span><span class="z-source z-java"><span class="z-meta z-import z-java"><span class="z-keyword z-control z-import z-java">import</span> <span class="z-meta z-path z-java"><span class="z-support z-type z-package z-java">org</span></span><span class="z-meta z-path z-java"><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">apache</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">flink</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">streaming</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">api</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">functions</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-class z-import z-java">KeyedProcessFunction</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span><span class="z-source z-java"><span class="z-meta z-import z-java"><span class="z-keyword z-control z-import z-java">import</span> <span class="z-meta z-path z-java"><span class="z-support z-type z-package z-java">org</span></span><span class="z-meta z-path z-java"><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">apache</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">flink</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">streaming</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">api</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">functions</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">windowing</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-class z-import z-java">ProcessWindowFunction</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span><span class="z-source z-java"><span class="z-meta z-import z-java"><span class="z-keyword z-control z-import z-java">import</span> <span class="z-meta z-path z-java"><span class="z-support z-type z-package z-java">org</span></span><span class="z-meta z-path z-java"><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">apache</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">flink</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">streaming</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">api</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">windowing</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">assigners</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-class z-import z-java">TumblingEventTimeWindows</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span><span class="z-source z-java"><span class="z-meta z-import z-java"><span class="z-keyword z-control z-import z-java">import</span> <span class="z-meta z-path z-java"><span class="z-support z-type z-package z-java">org</span></span><span class="z-meta z-path z-java"><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">apache</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">flink</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">streaming</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">api</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">windowing</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">time</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-class z-import z-java">Time</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span><span class="z-source z-java"><span class="z-meta z-import z-java"><span class="z-keyword z-control z-import z-java">import</span> <span class="z-meta z-path z-java"><span class="z-support z-type z-package z-java">org</span></span><span class="z-meta z-path z-java"><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">apache</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">flink</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">streaming</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">api</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">windowing</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">windows</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-class z-import z-java">TimeWindow</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span><span class="z-source z-java"><span class="z-meta z-import z-java"><span class="z-keyword z-control z-import z-java">import</span> <span class="z-meta z-path z-java"><span class="z-support z-type z-package z-java">org</span></span><span class="z-meta z-path z-java"><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">apache</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">flink</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">streaming</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">connectors</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">kafka</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-class z-import z-java">FlinkKafkaConsumer</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span><span class="z-source z-java"><span class="z-meta z-import z-java"><span class="z-keyword z-control z-import z-java">import</span> <span class="z-meta z-path z-java"><span class="z-support z-type z-package z-java">org</span></span><span class="z-meta z-path z-java"><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">apache</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">flink</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">streaming</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">connectors</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">kafka</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-class z-import z-java">FlinkKafkaProducer</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span><span class="z-source z-java"><span class="z-meta z-import z-java"><span class="z-keyword z-control z-import z-java">import</span> <span class="z-meta z-path z-java"><span class="z-support z-type z-package z-java">org</span></span><span class="z-meta z-path z-java"><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">apache</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">flink</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">util</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-class z-import z-java">Collector</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span><span class="z-source z-java"> </span><span class="z-source z-java"><span class="z-meta z-import z-java"><span class="z-keyword z-control z-import z-java">import</span> <span class="z-meta z-path z-java"><span class="z-support z-type z-package z-java">java</span></span><span class="z-meta z-path z-java"><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">time</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-class z-import z-java">Duration</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span><span class="z-source z-java"><span class="z-meta z-import z-java"><span class="z-keyword z-control z-import z-java">import</span> <span class="z-meta z-path z-java"><span class="z-support z-type z-package z-java">java</span></span><span class="z-meta z-path z-java"><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-type z-package z-java">util</span><span class="z-punctuation z-accessor z-dot z-java">.</span></span><span class="z-meta z-path z-java"><span class="z-keyword z-operator z-wildcard z-asterisk z-java">*</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span><span class="z-source z-java"> </span><span class="z-source z-java"><span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Transaction POJO </span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-storage z-modifier z-java">public</span> <span class="z-meta z-class z-identifier z-java"><span class="z-storage z-type z-java">class</span> <span class="z-entity z-name z-class z-java">Transaction</span></span> <span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-punctuation z-section z-block z-begin z-java">{</span> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-storage z-modifier z-java">public</span> <span class="z-support z-class z-java">String</span> <span class="z-meta z-field z-java">transactionId</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-storage z-modifier z-java">public</span> <span class="z-support z-class z-java">String</span> <span class="z-meta z-field z-java">userId</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-storage z-modifier z-java">public</span> <span class="z-support z-class z-java">String</span> <span class="z-meta z-field z-java">cardNumber</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-storage z-modifier z-java">public</span> <span class="z-storage z-type z-primitive z-java">double</span> <span class="z-meta z-field z-java">amount</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-storage z-modifier z-java">public</span> <span class="z-support z-class z-java">String</span> <span class="z-meta z-field z-java">merchantId</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-storage z-modifier z-java">public</span> <span class="z-support z-class z-java">String</span> <span class="z-meta z-field z-java">merchantCategory</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-storage z-modifier z-java">public</span> <span class="z-support z-class z-java">String</span> <span class="z-meta z-field z-java">location</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-storage z-modifier z-java">public</span> <span class="z-storage z-type z-primitive z-java">double</span> <span class="z-meta z-field z-java">latitude</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-storage z-modifier z-java">public</span> <span class="z-storage z-type z-primitive z-java">double</span> <span class="z-meta z-field z-java">longitude</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-storage z-modifier z-java">public</span> <span class="z-storage z-type z-primitive z-java">long</span> <span class="z-meta z-field z-java">timestamp</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Constructor, getters, setters </span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-punctuation z-section z-block z-end z-java">}</span></span></span></span> </span><span class="z-source z-java"> </span><span class="z-source z-java"><span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Fraud Alert POJO </span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-storage z-modifier z-java">public</span> <span class="z-meta z-class z-identifier z-java"><span class="z-storage z-type z-java">class</span> <span class="z-entity z-name z-class z-java">FraudAlert</span></span> <span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-punctuation z-section z-block z-begin z-java">{</span> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-storage z-modifier z-java">public</span> <span class="z-support z-class z-java">String</span> <span class="z-meta z-field z-java">transactionId</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-storage z-modifier z-java">public</span> <span class="z-support z-class z-java">String</span> <span class="z-meta z-field z-java">userId</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-storage z-modifier z-java">public</span> <span class="z-support z-class z-java">String</span> <span class="z-meta z-field z-java">alertType</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-storage z-modifier z-java">public</span> <span class="z-support z-class z-java">String</span> <span class="z-meta z-field z-java">reason</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-storage z-modifier z-java">public</span> <span class="z-storage z-type z-primitive z-java">double</span> <span class="z-meta z-field z-java">riskScore</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-storage z-modifier z-java">public</span> <span class="z-storage z-type z-primitive z-java">long</span> <span class="z-meta z-field z-java">timestamp</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Constructor, getters, setters </span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-punctuation z-section z-block z-end z-java">}</span></span></span></span> </span><span class="z-source z-java"> </span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-storage z-modifier z-java">public</span> <span class="z-meta z-class z-identifier z-java"><span class="z-storage z-type z-java">class</span> <span class="z-entity z-name z-class z-java">RealTimeFraudDetection</span></span> <span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-punctuation z-section z-block z-begin z-java">{</span> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-storage z-modifier z-java">public</span> <span class="z-storage z-modifier z-java">static</span> <span class="z-meta z-method z-java"><span class="z-storage z-type z-void z-java">void</span> <span class="z-meta z-method z-identifier z-java"><span class="z-entity z-name z-function z-java">main</span></span><span class="z-meta z-method z-parameters z-java"><span class="z-meta z-parens z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-support z-class z-java">String</span><span class="z-storage z-modifier z-array z-java">[]</span> <span class="z-variable z-parameter z-java">args</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span> <span class="z-meta z-method z-throws z-java"><span class="z-keyword z-declaration z-throws z-java">throws</span> <span class="z-support z-class z-java">Exception</span> </span><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-punctuation z-section z-block z-begin z-java">{</span></span></span></span><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Set up the streaming execution environment </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-support z-class z-java">StreamExecutionEnvironment</span> env <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-support z-class z-java">StreamExecutionEnvironment</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">getExecutionEnvironment</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Enable checkpointing for fault tolerance (every 10 seconds) </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> env<span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">enableCheckpointing</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-constant z-numeric z-integer z-decimal z-java">10000</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> env<span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">getCheckpointConfig</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">setCheckpointingMode</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-support z-class z-java">CheckpointingMode</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-constant z-other z-java">EXACTLY_ONCE</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Configure Kafka consumer </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-support z-class z-java">Properties</span> kafkaProps <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-meta z-instantiation z-java"><span class="z-keyword z-other z-storage z-new z-java">new</span> <span class="z-support z-class z-java">Properties</span><span class="z-meta z-parens z-constructor-arguments z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> kafkaProps<span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">setProperty</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-string z-quoted z-double z-java"><span class="z-punctuation z-definition z-string z-begin z-java">&quot;</span>bootstrap.servers<span class="z-punctuation z-definition z-string z-end z-java">&quot;</span></span><span class="z-punctuation z-separator z-comma z-java">,</span> <span class="z-string z-quoted z-double z-java"><span class="z-punctuation z-definition z-string z-begin z-java">&quot;</span>localhost:9092<span class="z-punctuation z-definition z-string z-end z-java">&quot;</span></span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> kafkaProps<span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">setProperty</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-string z-quoted z-double z-java"><span class="z-punctuation z-definition z-string z-begin z-java">&quot;</span>group.id<span class="z-punctuation z-definition z-string z-end z-java">&quot;</span></span><span class="z-punctuation z-separator z-comma z-java">,</span> <span class="z-string z-quoted z-double z-java"><span class="z-punctuation z-definition z-string z-begin z-java">&quot;</span>fraud-detection<span class="z-punctuation z-definition z-string z-end z-java">&quot;</span></span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Create Kafka source for transactions </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-support z-class z-java">FlinkKafkaConsumer</span><span class="z-meta z-generic z-java"><span class="z-punctuation z-definition z-generic z-begin z-java">&lt;</span><span class="z-support z-class z-java">Transaction</span><span class="z-punctuation z-definition z-generic z-end z-java">&gt;</span></span> kafkaConsumer <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-meta z-instantiation z-java"><span class="z-keyword z-other z-storage z-new z-java">new</span> <span class="z-support z-class z-java">FlinkKafkaConsumer</span><span class="z-punctuation z-definition z-generic z-diamond z-java">&lt;&gt;</span><span class="z-meta z-parens z-constructor-arguments z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-assignment z-rhs z-java"><span class="z-meta z-instantiation z-java"><span class="z-meta z-parens z-constructor-arguments z-java"> <span class="z-string z-quoted z-double z-java"><span class="z-punctuation z-definition z-string z-begin z-java">&quot;</span>transactions<span class="z-punctuation z-definition z-string z-end z-java">&quot;</span></span><span class="z-punctuation z-separator z-comma z-java">,</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-assignment z-rhs z-java"><span class="z-meta z-instantiation z-java"><span class="z-meta z-parens z-constructor-arguments z-java"> <span class="z-meta z-instantiation z-java"><span class="z-keyword z-other z-storage z-new z-java">new</span> <span class="z-support z-class z-java">TransactionDeserializationSchema</span><span class="z-meta z-parens z-constructor-arguments z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-separator z-comma z-java">,</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-assignment z-rhs z-java"><span class="z-meta z-instantiation z-java"><span class="z-meta z-parens z-constructor-arguments z-java"> kafkaProps </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-assignment z-rhs z-java"><span class="z-meta z-instantiation z-java"><span class="z-meta z-parens z-constructor-arguments z-java"> <span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Assign watermarks for event time processing </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-support z-class z-java">WatermarkStrategy</span><span class="z-meta z-generic z-java"><span class="z-punctuation z-definition z-generic z-begin z-java">&lt;</span><span class="z-support z-class z-java">Transaction</span><span class="z-punctuation z-definition z-generic z-end z-java">&gt;</span></span> watermarkStrategy <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-support z-class z-java">WatermarkStrategy</span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-assignment z-rhs z-java"> <span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-generic z-java"><span class="z-punctuation z-definition z-generic z-begin z-java">&lt;</span><span class="z-support z-class z-java">Transaction</span><span class="z-punctuation z-definition z-generic z-end z-java">&gt;</span></span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">forBoundedOutOfOrderness</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-support z-class z-java">Duration</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">ofSeconds</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-constant z-numeric z-integer z-decimal z-java">5</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-assignment z-rhs z-java"> <span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">withTimestampAssigner</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-meta z-function z-anonymous z-parameters z-java"><span class="z-meta z-function z-anonymous z-parameters z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span></span></span><span class="z-meta z-function z-anonymous z-parameters z-java"><span class="z-variable z-parameter z-java">transaction</span><span class="z-punctuation z-separator z-comma z-java">,</span> <span class="z-variable z-parameter z-java">timestamp</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span> <span class="z-meta z-function z-anonymous z-body z-java"><span class="z-storage z-type z-function z-anonymous z-java">-&gt;</span> transaction<span class="z-punctuation z-accessor z-dot z-java">.</span>timestamp</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Read transaction stream </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-support z-class z-java">DataStream</span><span class="z-meta z-generic z-java"><span class="z-punctuation z-definition z-generic z-begin z-java">&lt;</span><span class="z-support z-class z-java">Transaction</span><span class="z-punctuation z-definition z-generic z-end z-java">&gt;</span></span> transactions <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> env </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-assignment z-rhs z-java"> <span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">addSource</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span>kafkaConsumer<span class="z-punctuation z-section z-parens z-end z-java">)</span></span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-assignment z-rhs z-java"> <span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">assignTimestampsAndWatermarks</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span>watermarkStrategy<span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Rule 1: Detect high-velocity transactions (&gt;5 transactions in 1 minute) </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-support z-class z-java">DataStream</span><span class="z-meta z-generic z-java"><span class="z-punctuation z-definition z-generic z-begin z-java">&lt;</span><span class="z-support z-class z-java">FraudAlert</span><span class="z-punctuation z-definition z-generic z-end z-java">&gt;</span></span> velocityAlerts <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> transactions </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-assignment z-rhs z-java"> <span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">keyBy</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-meta z-function z-anonymous z-parameters z-java"><span class="z-variable z-parameter z-java">t</span></span> <span class="z-meta z-function z-anonymous z-body z-java"><span class="z-storage z-type z-function z-anonymous z-java">-&gt;</span> t<span class="z-punctuation z-accessor z-dot z-java">.</span>userId</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-assignment z-rhs z-java"> <span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">window</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-support z-class z-java">TumblingEventTimeWindows</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">of</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-support z-class z-java">Time</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">minutes</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-constant z-numeric z-integer z-decimal z-java">1</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-assignment z-rhs z-java"> <span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">process</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-meta z-instantiation z-java"><span class="z-keyword z-other z-storage z-new z-java">new</span> <span class="z-support z-class z-java">VelocityDetector</span><span class="z-meta z-parens z-constructor-arguments z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Rule 2: Detect unusual amount patterns </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-support z-class z-java">DataStream</span><span class="z-meta z-generic z-java"><span class="z-punctuation z-definition z-generic z-begin z-java">&lt;</span><span class="z-support z-class z-java">FraudAlert</span><span class="z-punctuation z-definition z-generic z-end z-java">&gt;</span></span> amountAlerts <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> transactions </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-assignment z-rhs z-java"> <span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">keyBy</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-meta z-function z-anonymous z-parameters z-java"><span class="z-variable z-parameter z-java">t</span></span> <span class="z-meta z-function z-anonymous z-body z-java"><span class="z-storage z-type z-function z-anonymous z-java">-&gt;</span> t<span class="z-punctuation z-accessor z-dot z-java">.</span>userId</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-assignment z-rhs z-java"> <span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">process</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-meta z-instantiation z-java"><span class="z-keyword z-other z-storage z-new z-java">new</span> <span class="z-support z-class z-java">UnusualAmountDetector</span><span class="z-meta z-parens z-constructor-arguments z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Rule 3: Detect rapid location changes </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-support z-class z-java">DataStream</span><span class="z-meta z-generic z-java"><span class="z-punctuation z-definition z-generic z-begin z-java">&lt;</span><span class="z-support z-class z-java">FraudAlert</span><span class="z-punctuation z-definition z-generic z-end z-java">&gt;</span></span> locationAlerts <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> transactions </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-assignment z-rhs z-java"> <span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">keyBy</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-meta z-function z-anonymous z-parameters z-java"><span class="z-variable z-parameter z-java">t</span></span> <span class="z-meta z-function z-anonymous z-body z-java"><span class="z-storage z-type z-function z-anonymous z-java">-&gt;</span> t<span class="z-punctuation z-accessor z-dot z-java">.</span>userId</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-assignment z-rhs z-java"> <span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">process</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-meta z-instantiation z-java"><span class="z-keyword z-other z-storage z-new z-java">new</span> <span class="z-support z-class z-java">LocationAnomalyDetector</span><span class="z-meta z-parens z-constructor-arguments z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Rule 4: Detect suspicious transaction sequences </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-support z-class z-java">DataStream</span><span class="z-meta z-generic z-java"><span class="z-punctuation z-definition z-generic z-begin z-java">&lt;</span><span class="z-support z-class z-java">FraudAlert</span><span class="z-punctuation z-definition z-generic z-end z-java">&gt;</span></span> sequenceAlerts <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> transactions </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-assignment z-rhs z-java"> <span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">keyBy</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-meta z-function z-anonymous z-parameters z-java"><span class="z-variable z-parameter z-java">t</span></span> <span class="z-meta z-function z-anonymous z-body z-java"><span class="z-storage z-type z-function z-anonymous z-java">-&gt;</span> t<span class="z-punctuation z-accessor z-dot z-java">.</span>userId</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-assignment z-rhs z-java"> <span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">process</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-meta z-instantiation z-java"><span class="z-keyword z-other z-storage z-new z-java">new</span> <span class="z-support z-class z-java">SuspiciousSequenceDetector</span><span class="z-meta z-parens z-constructor-arguments z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Combine all alerts </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-support z-class z-java">DataStream</span><span class="z-meta z-generic z-java"><span class="z-punctuation z-definition z-generic z-begin z-java">&lt;</span><span class="z-support z-class z-java">FraudAlert</span><span class="z-punctuation z-definition z-generic z-end z-java">&gt;</span></span> allAlerts <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> velocityAlerts </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-assignment z-rhs z-java"> <span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">union</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span>amountAlerts<span class="z-punctuation z-section z-parens z-end z-java">)</span></span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-assignment z-rhs z-java"> <span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">union</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span>locationAlerts<span class="z-punctuation z-section z-parens z-end z-java">)</span></span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-assignment z-rhs z-java"> <span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">union</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span>sequenceAlerts<span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Deduplicate and enrich alerts </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-support z-class z-java">DataStream</span><span class="z-meta z-generic z-java"><span class="z-punctuation z-definition z-generic z-begin z-java">&lt;</span><span class="z-support z-class z-java">FraudAlert</span><span class="z-punctuation z-definition z-generic z-end z-java">&gt;</span></span> enrichedAlerts <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> allAlerts </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-assignment z-rhs z-java"> <span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">keyBy</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-meta z-function z-anonymous z-parameters z-java"><span class="z-variable z-parameter z-java">alert</span></span> <span class="z-meta z-function z-anonymous z-body z-java"><span class="z-storage z-type z-function z-anonymous z-java">-&gt;</span> alert<span class="z-punctuation z-accessor z-dot z-java">.</span>transactionId</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-assignment z-rhs z-java"> <span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">window</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-support z-class z-java">TumblingEventTimeWindows</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">of</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-support z-class z-java">Time</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">seconds</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-constant z-numeric z-integer z-decimal z-java">5</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-assignment z-rhs z-java"> <span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">process</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-meta z-instantiation z-java"><span class="z-keyword z-other z-storage z-new z-java">new</span> <span class="z-support z-class z-java">AlertAggregator</span><span class="z-meta z-parens z-constructor-arguments z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Send high-risk alerts to Kafka </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-support z-class z-java">FlinkKafkaProducer</span><span class="z-meta z-generic z-java"><span class="z-punctuation z-definition z-generic z-begin z-java">&lt;</span><span class="z-support z-class z-java">FraudAlert</span><span class="z-punctuation z-definition z-generic z-end z-java">&gt;</span></span> kafkaProducer <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-meta z-instantiation z-java"><span class="z-keyword z-other z-storage z-new z-java">new</span> <span class="z-support z-class z-java">FlinkKafkaProducer</span><span class="z-punctuation z-definition z-generic z-diamond z-java">&lt;&gt;</span><span class="z-meta z-parens z-constructor-arguments z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-assignment z-rhs z-java"><span class="z-meta z-instantiation z-java"><span class="z-meta z-parens z-constructor-arguments z-java"> <span class="z-string z-quoted z-double z-java"><span class="z-punctuation z-definition z-string z-begin z-java">&quot;</span>fraud-alerts<span class="z-punctuation z-definition z-string z-end z-java">&quot;</span></span><span class="z-punctuation z-separator z-comma z-java">,</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-assignment z-rhs z-java"><span class="z-meta z-instantiation z-java"><span class="z-meta z-parens z-constructor-arguments z-java"> <span class="z-meta z-instantiation z-java"><span class="z-keyword z-other z-storage z-new z-java">new</span> <span class="z-support z-class z-java">FraudAlertSerializationSchema</span><span class="z-meta z-parens z-constructor-arguments z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-separator z-comma z-java">,</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-assignment z-rhs z-java"><span class="z-meta z-instantiation z-java"><span class="z-meta z-parens z-constructor-arguments z-java"> kafkaProps<span class="z-punctuation z-separator z-comma z-java">,</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-assignment z-rhs z-java"><span class="z-meta z-instantiation z-java"><span class="z-meta z-parens z-constructor-arguments z-java"> <span class="z-support z-class z-java">FlinkKafkaProducer</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-support z-class z-java">Semantic</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-constant z-other z-java">EXACTLY_ONCE</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-assignment z-rhs z-java"><span class="z-meta z-instantiation z-java"><span class="z-meta z-parens z-constructor-arguments z-java"> <span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> enrichedAlerts </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">filter</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-meta z-function z-anonymous z-parameters z-java"><span class="z-variable z-parameter z-java">alert</span></span> <span class="z-meta z-function z-anonymous z-body z-java"><span class="z-storage z-type z-function z-anonymous z-java">-&gt;</span> alert<span class="z-punctuation z-accessor z-dot z-java">.</span>riskScore <span class="z-keyword z-operator z-comparison z-java">&gt;</span> <span class="z-constant z-numeric z-float z-decimal z-java">0<span class="z-punctuation z-separator z-decimal z-java">.</span>7</span></span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">addSink</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span>kafkaProducer<span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Store all alerts in database for analysis </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> enrichedAlerts<span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">addSink</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-meta z-instantiation z-java"><span class="z-keyword z-other z-storage z-new z-java">new</span> <span class="z-support z-class z-java">JdbcSink</span><span class="z-meta z-parens z-constructor-arguments z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Execute the Flink job </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> env<span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">execute</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-string z-quoted z-double z-java"><span class="z-punctuation z-definition z-string z-begin z-java">&quot;</span>Real-Time Fraud Detection<span class="z-punctuation z-definition z-string z-end z-java">&quot;</span></span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-punctuation z-section z-block z-end z-java">}</span></span></span> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Velocity Detector: Flags if user makes &gt;5 transactions in 1 minute </span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-meta z-class z-java"><span class="z-storage z-modifier z-java">public</span> <span class="z-storage z-modifier z-java">static</span> <span class="z-meta z-class z-identifier z-java"><span class="z-storage z-type z-java">class</span> <span class="z-entity z-name z-class z-java">VelocityDetector</span></span> </span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"> <span class="z-meta z-class z-extends z-java"><span class="z-keyword z-declaration z-extends z-java">extends</span> <span class="z-entity z-other z-inherited-class z-java">ProcessWindowFunction</span><span class="z-meta z-generic z-java"><span class="z-punctuation z-definition z-generic z-begin z-java">&lt;</span><span class="z-support z-class z-java">Transaction</span><span class="z-punctuation z-separator z-comma z-java">,</span> <span class="z-support z-class z-java">FraudAlert</span><span class="z-punctuation z-separator z-comma z-java">,</span> <span class="z-support z-class z-java">String</span><span class="z-punctuation z-separator z-comma z-java">,</span> <span class="z-support z-class z-java">TimeWindow</span><span class="z-punctuation z-definition z-generic z-end z-java">&gt;</span></span> </span><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-punctuation z-section z-block z-begin z-java">{</span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-meta z-annotation z-java"><span class="z-punctuation z-definition z-annotation z-java">@</span></span><span class="z-meta z-annotation z-java"><span class="z-meta z-annotation z-identifier z-java"><span class="z-variable z-annotation z-java">Override</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-annotation z-java"><span class="z-meta z-annotation z-identifier z-java"> </span></span><span class="z-storage z-modifier z-java">public</span> <span class="z-meta z-method z-java"><span class="z-storage z-type z-void z-java">void</span> <span class="z-meta z-method z-identifier z-java"><span class="z-entity z-name z-function z-java">process</span></span><span class="z-meta z-method z-parameters z-java"><span class="z-meta z-parens z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-support z-class z-java">String</span> <span class="z-variable z-parameter z-java">userId</span><span class="z-punctuation z-separator z-comma z-java">,</span> <span class="z-support z-class z-java">Context</span> <span class="z-variable z-parameter z-java">context</span><span class="z-punctuation z-separator z-comma z-java">,</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-parameters z-java"><span class="z-meta z-parens z-java"> <span class="z-support z-class z-java">Iterable</span><span class="z-meta z-generic z-java"><span class="z-punctuation z-definition z-generic z-begin z-java">&lt;</span><span class="z-support z-class z-java">Transaction</span><span class="z-punctuation z-definition z-generic z-end z-java">&gt;</span></span> <span class="z-variable z-parameter z-java">transactions</span><span class="z-punctuation z-separator z-comma z-java">,</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-parameters z-java"><span class="z-meta z-parens z-java"> <span class="z-support z-class z-java">Collector</span><span class="z-meta z-generic z-java"><span class="z-punctuation z-definition z-generic z-begin z-java">&lt;</span><span class="z-support z-class z-java">FraudAlert</span><span class="z-punctuation z-definition z-generic z-end z-java">&gt;</span></span> <span class="z-variable z-parameter z-java">out</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span> <span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-punctuation z-section z-block z-begin z-java">{</span></span></span></span><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-storage z-type z-primitive z-java">int</span> count <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-constant z-numeric z-integer z-decimal z-java">0</span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-support z-class z-java">Transaction</span> lastTxn <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-constant z-language z-java">null</span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-keyword z-control z-loop z-for z-java">for</span> <span class="z-meta z-parens z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-support z-class z-java">Transaction</span> txn <span class="z-keyword z-operator z-ternary z-java">:</span> transactions<span class="z-punctuation z-section z-parens z-end z-java">)</span></span> <span class="z-meta z-block z-java"><span class="z-punctuation z-section z-block z-begin z-java">{</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> count<span class="z-keyword z-operator z-increment-decrement z-java">++</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> lastTxn <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> txn</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> <span class="z-punctuation z-section z-block z-end z-java">}</span></span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Alert if more than 5 transactions in 1 minute </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-keyword z-control z-conditional z-if z-java">if</span> <span class="z-meta z-parens z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span>count <span class="z-keyword z-operator z-comparison z-java">&gt;</span> <span class="z-constant z-numeric z-integer z-decimal z-java">5</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span> <span class="z-meta z-block z-java"><span class="z-punctuation z-section z-block z-begin z-java">{</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> <span class="z-support z-class z-java">FraudAlert</span> alert <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-meta z-instantiation z-java"><span class="z-keyword z-other z-storage z-new z-java">new</span> <span class="z-support z-class z-java">FraudAlert</span><span class="z-meta z-parens z-constructor-arguments z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> alert<span class="z-punctuation z-accessor z-dot z-java">.</span>transactionId <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> lastTxn<span class="z-punctuation z-accessor z-dot z-java">.</span>transactionId</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> alert<span class="z-punctuation z-accessor z-dot z-java">.</span>userId <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> userId</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> alert<span class="z-punctuation z-accessor z-dot z-java">.</span>alertType <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-string z-quoted z-double z-java"><span class="z-punctuation z-definition z-string z-begin z-java">&quot;</span>HIGH_VELOCITY<span class="z-punctuation z-definition z-string z-end z-java">&quot;</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> alert<span class="z-punctuation z-accessor z-dot z-java">.</span>reason <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-support z-class z-java">String</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">format</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-string z-quoted z-double z-java"><span class="z-punctuation z-definition z-string z-begin z-java">&quot;</span>%d transactions in 1 minute<span class="z-punctuation z-definition z-string z-end z-java">&quot;</span></span><span class="z-punctuation z-separator z-comma z-java">,</span> count<span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> alert<span class="z-punctuation z-accessor z-dot z-java">.</span>riskScore <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-support z-class z-java">Math</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">min</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-constant z-numeric z-float z-decimal z-java">0<span class="z-punctuation z-separator z-decimal z-java">.</span>5</span> <span class="z-keyword z-operator z-arithmetic z-java">+</span> <span class="z-meta z-parens z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span>count <span class="z-keyword z-operator z-arithmetic z-java">-</span> <span class="z-constant z-numeric z-integer z-decimal z-java">5</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span> <span class="z-keyword z-operator z-arithmetic z-java">*</span> <span class="z-constant z-numeric z-float z-decimal z-java">0<span class="z-punctuation z-separator z-decimal z-java">.</span>1</span><span class="z-punctuation z-separator z-comma z-java">,</span> <span class="z-constant z-numeric z-float z-decimal z-java">1<span class="z-punctuation z-separator z-decimal z-java">.</span>0</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> alert<span class="z-punctuation z-accessor z-dot z-java">.</span>timestamp <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-support z-class z-java">System</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">currentTimeMillis</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> out<span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">collect</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span>alert<span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> <span class="z-punctuation z-section z-block z-end z-java">}</span></span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-punctuation z-section z-block z-end z-java">}</span></span></span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-punctuation z-section z-block z-end z-java">}</span></span></span></span> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Unusual Amount Detector: Compares transaction with user&#39;s historical patterns </span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-meta z-class z-java"><span class="z-storage z-modifier z-java">public</span> <span class="z-storage z-modifier z-java">static</span> <span class="z-meta z-class z-identifier z-java"><span class="z-storage z-type z-java">class</span> <span class="z-entity z-name z-class z-java">UnusualAmountDetector</span></span> </span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"> <span class="z-meta z-class z-extends z-java"><span class="z-keyword z-declaration z-extends z-java">extends</span> <span class="z-entity z-other z-inherited-class z-java">KeyedProcessFunction</span><span class="z-meta z-generic z-java"><span class="z-punctuation z-definition z-generic z-begin z-java">&lt;</span><span class="z-support z-class z-java">String</span><span class="z-punctuation z-separator z-comma z-java">,</span> <span class="z-support z-class z-java">Transaction</span><span class="z-punctuation z-separator z-comma z-java">,</span> <span class="z-support z-class z-java">FraudAlert</span><span class="z-punctuation z-definition z-generic z-end z-java">&gt;</span></span> </span><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-punctuation z-section z-block z-begin z-java">{</span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> State to store user&#39;s transaction history </span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-storage z-modifier z-java">private</span> <span class="z-storage z-modifier z-java">transient</span> <span class="z-support z-class z-java">ValueState</span><span class="z-meta z-generic z-java"><span class="z-punctuation z-definition z-generic z-begin z-java">&lt;</span><span class="z-support z-class z-java">TransactionStats</span><span class="z-punctuation z-definition z-generic z-end z-java">&gt;</span></span> <span class="z-meta z-field z-java">statsState</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-meta z-annotation z-java"><span class="z-punctuation z-definition z-annotation z-java">@</span></span><span class="z-meta z-annotation z-java"><span class="z-meta z-annotation z-identifier z-java"><span class="z-variable z-annotation z-java">Override</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-annotation z-java"><span class="z-meta z-annotation z-identifier z-java"> </span></span><span class="z-storage z-modifier z-java">public</span> <span class="z-meta z-method z-java"><span class="z-storage z-type z-void z-java">void</span> <span class="z-meta z-method z-identifier z-java"><span class="z-entity z-name z-function z-java">open</span></span><span class="z-meta z-method z-parameters z-java"><span class="z-meta z-parens z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-support z-class z-java">Configuration</span> <span class="z-variable z-parameter z-java">parameters</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span> <span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-punctuation z-section z-block z-begin z-java">{</span></span></span></span><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-support z-class z-java">ValueStateDescriptor</span><span class="z-meta z-generic z-java"><span class="z-punctuation z-definition z-generic z-begin z-java">&lt;</span><span class="z-support z-class z-java">TransactionStats</span><span class="z-punctuation z-definition z-generic z-end z-java">&gt;</span></span> descriptor <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-assignment z-rhs z-java"> <span class="z-meta z-instantiation z-java"><span class="z-keyword z-other z-storage z-new z-java">new</span> <span class="z-support z-class z-java">ValueStateDescriptor</span><span class="z-punctuation z-definition z-generic z-diamond z-java">&lt;&gt;</span><span class="z-meta z-parens z-constructor-arguments z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-string z-quoted z-double z-java"><span class="z-punctuation z-definition z-string z-begin z-java">&quot;</span>transaction-stats<span class="z-punctuation z-definition z-string z-end z-java">&quot;</span></span><span class="z-punctuation z-separator z-comma z-java">,</span> <span class="z-support z-class z-java">TransactionStats</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-variable z-language z-java">class</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> statsState <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">getRuntimeContext</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">getState</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span>descriptor<span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-punctuation z-section z-block z-end z-java">}</span></span></span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-meta z-annotation z-java"><span class="z-punctuation z-definition z-annotation z-java">@</span></span><span class="z-meta z-annotation z-java"><span class="z-meta z-annotation z-identifier z-java"><span class="z-variable z-annotation z-java">Override</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-annotation z-java"><span class="z-meta z-annotation z-identifier z-java"> </span></span><span class="z-storage z-modifier z-java">public</span> <span class="z-meta z-method z-java"><span class="z-storage z-type z-void z-java">void</span> <span class="z-meta z-method z-identifier z-java"><span class="z-entity z-name z-function z-java">processElement</span></span><span class="z-meta z-method z-parameters z-java"><span class="z-meta z-parens z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-support z-class z-java">Transaction</span> <span class="z-variable z-parameter z-java">txn</span><span class="z-punctuation z-separator z-comma z-java">,</span> <span class="z-support z-class z-java">Context</span> <span class="z-variable z-parameter z-java">ctx</span><span class="z-punctuation z-separator z-comma z-java">,</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-parameters z-java"><span class="z-meta z-parens z-java"> <span class="z-support z-class z-java">Collector</span><span class="z-meta z-generic z-java"><span class="z-punctuation z-definition z-generic z-begin z-java">&lt;</span><span class="z-support z-class z-java">FraudAlert</span><span class="z-punctuation z-definition z-generic z-end z-java">&gt;</span></span> <span class="z-variable z-parameter z-java">out</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span> <span class="z-meta z-method z-throws z-java"><span class="z-keyword z-declaration z-throws z-java">throws</span> <span class="z-support z-class z-java">Exception</span> </span><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-punctuation z-section z-block z-begin z-java">{</span></span></span></span><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-support z-class z-java">TransactionStats</span> stats <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> statsState<span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">value</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-keyword z-control z-conditional z-if z-java">if</span> <span class="z-meta z-parens z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span>stats <span class="z-keyword z-operator z-comparison z-java">==</span> <span class="z-constant z-language z-java">null</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span> <span class="z-meta z-block z-java"><span class="z-punctuation z-section z-block z-begin z-java">{</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> stats <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-meta z-instantiation z-java"><span class="z-keyword z-other z-storage z-new z-java">new</span> <span class="z-support z-class z-java">TransactionStats</span><span class="z-meta z-parens z-constructor-arguments z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> <span class="z-punctuation z-section z-block z-end z-java">}</span></span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Check if amount is significantly higher than average </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-keyword z-control z-conditional z-if z-java">if</span> <span class="z-meta z-parens z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span>stats<span class="z-punctuation z-accessor z-dot z-java">.</span>count <span class="z-keyword z-operator z-comparison z-java">&gt;</span> <span class="z-constant z-numeric z-integer z-decimal z-java">10</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span> <span class="z-meta z-block z-java"><span class="z-punctuation z-section z-block z-begin z-java">{</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> <span class="z-storage z-type z-primitive z-java">double</span> avgAmount <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> stats<span class="z-punctuation z-accessor z-dot z-java">.</span>totalAmount <span class="z-keyword z-operator z-arithmetic z-java">/</span> stats<span class="z-punctuation z-accessor z-dot z-java">.</span>count</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> <span class="z-storage z-type z-primitive z-java">double</span> stdDev <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-support z-class z-java">Math</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">sqrt</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span>stats<span class="z-punctuation z-accessor z-dot z-java">.</span>sumOfSquares <span class="z-keyword z-operator z-arithmetic z-java">/</span> stats<span class="z-punctuation z-accessor z-dot z-java">.</span>count <span class="z-keyword z-operator z-arithmetic z-java">-</span> avgAmount <span class="z-keyword z-operator z-arithmetic z-java">*</span> avgAmount<span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Flag if amount is &gt; 3 standard deviations from mean </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> <span class="z-keyword z-control z-conditional z-if z-java">if</span> <span class="z-meta z-parens z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span>txn<span class="z-punctuation z-accessor z-dot z-java">.</span>amount <span class="z-keyword z-operator z-comparison z-java">&gt;</span> avgAmount <span class="z-keyword z-operator z-arithmetic z-java">+</span> <span class="z-meta z-parens z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-constant z-numeric z-integer z-decimal z-java">3</span> <span class="z-keyword z-operator z-arithmetic z-java">*</span> stdDev<span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span> <span class="z-meta z-block z-java"><span class="z-punctuation z-section z-block z-begin z-java">{</span> </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-block z-java"> <span class="z-support z-class z-java">FraudAlert</span> alert <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-meta z-instantiation z-java"><span class="z-keyword z-other z-storage z-new z-java">new</span> <span class="z-support z-class z-java">FraudAlert</span><span class="z-meta z-parens z-constructor-arguments z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-block z-java"> alert<span class="z-punctuation z-accessor z-dot z-java">.</span>transactionId <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> txn<span class="z-punctuation z-accessor z-dot z-java">.</span>transactionId</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-block z-java"> alert<span class="z-punctuation z-accessor z-dot z-java">.</span>userId <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> txn<span class="z-punctuation z-accessor z-dot z-java">.</span>userId</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-block z-java"> alert<span class="z-punctuation z-accessor z-dot z-java">.</span>alertType <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-string z-quoted z-double z-java"><span class="z-punctuation z-definition z-string z-begin z-java">&quot;</span>UNUSUAL_AMOUNT<span class="z-punctuation z-definition z-string z-end z-java">&quot;</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-block z-java"> alert<span class="z-punctuation z-accessor z-dot z-java">.</span>reason <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-support z-class z-java">String</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">format</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-string z-quoted z-double z-java"><span class="z-punctuation z-definition z-string z-begin z-java">&quot;</span>Amount $%.2f is %.1fx higher than average $%.2f<span class="z-punctuation z-definition z-string z-end z-java">&quot;</span></span><span class="z-punctuation z-separator z-comma z-java">,</span> </span></span></span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-block z-java"><span class="z-meta z-assignment z-rhs z-java"><span class="z-meta z-function-call z-java"> txn<span class="z-punctuation z-accessor z-dot z-java">.</span>amount<span class="z-punctuation z-separator z-comma z-java">,</span> txn<span class="z-punctuation z-accessor z-dot z-java">.</span>amount <span class="z-keyword z-operator z-arithmetic z-java">/</span> avgAmount<span class="z-punctuation z-separator z-comma z-java">,</span> avgAmount<span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-block z-java"> alert<span class="z-punctuation z-accessor z-dot z-java">.</span>riskScore <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-support z-class z-java">Math</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">min</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-meta z-parens z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span>txn<span class="z-punctuation z-accessor z-dot z-java">.</span>amount <span class="z-keyword z-operator z-arithmetic z-java">/</span> avgAmount <span class="z-keyword z-operator z-arithmetic z-java">-</span> <span class="z-constant z-numeric z-integer z-decimal z-java">1</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span> <span class="z-keyword z-operator z-arithmetic z-java">*</span> <span class="z-constant z-numeric z-float z-decimal z-java">0<span class="z-punctuation z-separator z-decimal z-java">.</span>2</span><span class="z-punctuation z-separator z-comma z-java">,</span> <span class="z-constant z-numeric z-float z-decimal z-java">0<span class="z-punctuation z-separator z-decimal z-java">.</span>9</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-block z-java"> alert<span class="z-punctuation z-accessor z-dot z-java">.</span>timestamp <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-support z-class z-java">System</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">currentTimeMillis</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-block z-java"> out<span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">collect</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span>alert<span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-block z-java"> <span class="z-punctuation z-section z-block z-end z-java">}</span></span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> <span class="z-punctuation z-section z-block z-end z-java">}</span></span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Update statistics </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> stats<span class="z-punctuation z-accessor z-dot z-java">.</span>count<span class="z-keyword z-operator z-increment-decrement z-java">++</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> stats<span class="z-punctuation z-accessor z-dot z-java">.</span>totalAmount <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">+=</span> txn<span class="z-punctuation z-accessor z-dot z-java">.</span>amount</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> stats<span class="z-punctuation z-accessor z-dot z-java">.</span>sumOfSquares <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">+=</span> txn<span class="z-punctuation z-accessor z-dot z-java">.</span>amount <span class="z-keyword z-operator z-arithmetic z-java">*</span> txn<span class="z-punctuation z-accessor z-dot z-java">.</span>amount</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> statsState<span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">update</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span>stats<span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-punctuation z-section z-block z-end z-java">}</span></span></span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-punctuation z-section z-block z-end z-java">}</span></span></span></span> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Location Anomaly Detector: Flags transactions from impossible locations </span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-meta z-class z-java"><span class="z-storage z-modifier z-java">public</span> <span class="z-storage z-modifier z-java">static</span> <span class="z-meta z-class z-identifier z-java"><span class="z-storage z-type z-java">class</span> <span class="z-entity z-name z-class z-java">LocationAnomalyDetector</span></span> </span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"> <span class="z-meta z-class z-extends z-java"><span class="z-keyword z-declaration z-extends z-java">extends</span> <span class="z-entity z-other z-inherited-class z-java">KeyedProcessFunction</span><span class="z-meta z-generic z-java"><span class="z-punctuation z-definition z-generic z-begin z-java">&lt;</span><span class="z-support z-class z-java">String</span><span class="z-punctuation z-separator z-comma z-java">,</span> <span class="z-support z-class z-java">Transaction</span><span class="z-punctuation z-separator z-comma z-java">,</span> <span class="z-support z-class z-java">FraudAlert</span><span class="z-punctuation z-definition z-generic z-end z-java">&gt;</span></span> </span><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-punctuation z-section z-block z-begin z-java">{</span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-storage z-modifier z-java">private</span> <span class="z-storage z-modifier z-java">transient</span> <span class="z-support z-class z-java">ValueState</span><span class="z-meta z-generic z-java"><span class="z-punctuation z-definition z-generic z-begin z-java">&lt;</span><span class="z-support z-class z-java">LastLocation</span><span class="z-punctuation z-definition z-generic z-end z-java">&gt;</span></span> <span class="z-meta z-field z-java">lastLocationState</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-meta z-annotation z-java"><span class="z-punctuation z-definition z-annotation z-java">@</span></span><span class="z-meta z-annotation z-java"><span class="z-meta z-annotation z-identifier z-java"><span class="z-variable z-annotation z-java">Override</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-annotation z-java"><span class="z-meta z-annotation z-identifier z-java"> </span></span><span class="z-storage z-modifier z-java">public</span> <span class="z-meta z-method z-java"><span class="z-storage z-type z-void z-java">void</span> <span class="z-meta z-method z-identifier z-java"><span class="z-entity z-name z-function z-java">open</span></span><span class="z-meta z-method z-parameters z-java"><span class="z-meta z-parens z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-support z-class z-java">Configuration</span> <span class="z-variable z-parameter z-java">parameters</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span> <span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-punctuation z-section z-block z-begin z-java">{</span></span></span></span><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-support z-class z-java">ValueStateDescriptor</span><span class="z-meta z-generic z-java"><span class="z-punctuation z-definition z-generic z-begin z-java">&lt;</span><span class="z-support z-class z-java">LastLocation</span><span class="z-punctuation z-definition z-generic z-end z-java">&gt;</span></span> descriptor <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-assignment z-rhs z-java"> <span class="z-meta z-instantiation z-java"><span class="z-keyword z-other z-storage z-new z-java">new</span> <span class="z-support z-class z-java">ValueStateDescriptor</span><span class="z-punctuation z-definition z-generic z-diamond z-java">&lt;&gt;</span><span class="z-meta z-parens z-constructor-arguments z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-string z-quoted z-double z-java"><span class="z-punctuation z-definition z-string z-begin z-java">&quot;</span>last-location<span class="z-punctuation z-definition z-string z-end z-java">&quot;</span></span><span class="z-punctuation z-separator z-comma z-java">,</span> <span class="z-support z-class z-java">LastLocation</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-variable z-language z-java">class</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> lastLocationState <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">getRuntimeContext</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">getState</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span>descriptor<span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-punctuation z-section z-block z-end z-java">}</span></span></span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-meta z-annotation z-java"><span class="z-punctuation z-definition z-annotation z-java">@</span></span><span class="z-meta z-annotation z-java"><span class="z-meta z-annotation z-identifier z-java"><span class="z-variable z-annotation z-java">Override</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-annotation z-java"><span class="z-meta z-annotation z-identifier z-java"> </span></span><span class="z-storage z-modifier z-java">public</span> <span class="z-meta z-method z-java"><span class="z-storage z-type z-void z-java">void</span> <span class="z-meta z-method z-identifier z-java"><span class="z-entity z-name z-function z-java">processElement</span></span><span class="z-meta z-method z-parameters z-java"><span class="z-meta z-parens z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-support z-class z-java">Transaction</span> <span class="z-variable z-parameter z-java">txn</span><span class="z-punctuation z-separator z-comma z-java">,</span> <span class="z-support z-class z-java">Context</span> <span class="z-variable z-parameter z-java">ctx</span><span class="z-punctuation z-separator z-comma z-java">,</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-parameters z-java"><span class="z-meta z-parens z-java"> <span class="z-support z-class z-java">Collector</span><span class="z-meta z-generic z-java"><span class="z-punctuation z-definition z-generic z-begin z-java">&lt;</span><span class="z-support z-class z-java">FraudAlert</span><span class="z-punctuation z-definition z-generic z-end z-java">&gt;</span></span> <span class="z-variable z-parameter z-java">out</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span> <span class="z-meta z-method z-throws z-java"><span class="z-keyword z-declaration z-throws z-java">throws</span> <span class="z-support z-class z-java">Exception</span> </span><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-punctuation z-section z-block z-begin z-java">{</span></span></span></span><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-support z-class z-java">LastLocation</span> lastLoc <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> lastLocationState<span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">value</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-keyword z-control z-conditional z-if z-java">if</span> <span class="z-meta z-parens z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span>lastLoc <span class="z-keyword z-operator z-comparison z-java">!=</span> <span class="z-constant z-language z-java">null</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span> <span class="z-meta z-block z-java"><span class="z-punctuation z-section z-block z-begin z-java">{</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Calculate distance between locations (using Haversine formula) </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> <span class="z-storage z-type z-primitive z-java">double</span> distance <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">calculateDistance</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span> </span></span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-assignment z-rhs z-java"><span class="z-meta z-function-call z-java"> lastLoc<span class="z-punctuation z-accessor z-dot z-java">.</span>latitude<span class="z-punctuation z-separator z-comma z-java">,</span> lastLoc<span class="z-punctuation z-accessor z-dot z-java">.</span>longitude<span class="z-punctuation z-separator z-comma z-java">,</span> </span></span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-assignment z-rhs z-java"><span class="z-meta z-function-call z-java"> txn<span class="z-punctuation z-accessor z-dot z-java">.</span>latitude<span class="z-punctuation z-separator z-comma z-java">,</span> txn<span class="z-punctuation z-accessor z-dot z-java">.</span>longitude </span></span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-assignment z-rhs z-java"><span class="z-meta z-function-call z-java"> <span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Calculate time difference in hours </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> <span class="z-storage z-type z-primitive z-java">double</span> timeDiffHours <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-meta z-parens z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span>txn<span class="z-punctuation z-accessor z-dot z-java">.</span>timestamp <span class="z-keyword z-operator z-arithmetic z-java">-</span> lastLoc<span class="z-punctuation z-accessor z-dot z-java">.</span>timestamp<span class="z-punctuation z-section z-parens z-end z-java">)</span></span> <span class="z-keyword z-operator z-arithmetic z-java">/</span> <span class="z-meta z-parens z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-constant z-numeric z-float z-decimal z-java">1000<span class="z-punctuation z-separator z-decimal z-java">.</span>0</span> <span class="z-keyword z-operator z-arithmetic z-java">*</span> <span class="z-constant z-numeric z-integer z-decimal z-java">3600</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Calculate required speed (km/h) </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> <span class="z-storage z-type z-primitive z-java">double</span> requiredSpeed <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> distance <span class="z-keyword z-operator z-arithmetic z-java">/</span> timeDiffHours</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Flag if speed &gt; 1000 km/h (impossible without flying) </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> <span class="z-keyword z-control z-conditional z-if z-java">if</span> <span class="z-meta z-parens z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span>timeDiffHours <span class="z-keyword z-operator z-comparison z-java">&lt;</span> <span class="z-constant z-numeric z-integer z-decimal z-java">2</span> <span class="z-keyword z-operator z-logical z-java">&amp;&amp;</span> requiredSpeed <span class="z-keyword z-operator z-comparison z-java">&gt;</span> <span class="z-constant z-numeric z-integer z-decimal z-java">1000</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span> <span class="z-meta z-block z-java"><span class="z-punctuation z-section z-block z-begin z-java">{</span> </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-block z-java"> <span class="z-support z-class z-java">FraudAlert</span> alert <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-meta z-instantiation z-java"><span class="z-keyword z-other z-storage z-new z-java">new</span> <span class="z-support z-class z-java">FraudAlert</span><span class="z-meta z-parens z-constructor-arguments z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-block z-java"> alert<span class="z-punctuation z-accessor z-dot z-java">.</span>transactionId <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> txn<span class="z-punctuation z-accessor z-dot z-java">.</span>transactionId</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-block z-java"> alert<span class="z-punctuation z-accessor z-dot z-java">.</span>userId <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> txn<span class="z-punctuation z-accessor z-dot z-java">.</span>userId</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-block z-java"> alert<span class="z-punctuation z-accessor z-dot z-java">.</span>alertType <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-string z-quoted z-double z-java"><span class="z-punctuation z-definition z-string z-begin z-java">&quot;</span>IMPOSSIBLE_TRAVEL<span class="z-punctuation z-definition z-string z-end z-java">&quot;</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-block z-java"> alert<span class="z-punctuation z-accessor z-dot z-java">.</span>reason <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-support z-class z-java">String</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">format</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-string z-quoted z-double z-java"><span class="z-punctuation z-definition z-string z-begin z-java">&quot;</span>%.0f km traveled in %.1f hours (%.0f km/h)<span class="z-punctuation z-definition z-string z-end z-java">&quot;</span></span><span class="z-punctuation z-separator z-comma z-java">,</span> </span></span></span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-block z-java"><span class="z-meta z-assignment z-rhs z-java"><span class="z-meta z-function-call z-java"> distance<span class="z-punctuation z-separator z-comma z-java">,</span> timeDiffHours<span class="z-punctuation z-separator z-comma z-java">,</span> requiredSpeed<span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-block z-java"> alert<span class="z-punctuation z-accessor z-dot z-java">.</span>riskScore <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-constant z-numeric z-float z-decimal z-java">0<span class="z-punctuation z-separator z-decimal z-java">.</span>95</span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-block z-java"> alert<span class="z-punctuation z-accessor z-dot z-java">.</span>timestamp <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-support z-class z-java">System</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">currentTimeMillis</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-block z-java"> out<span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">collect</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span>alert<span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-block z-java"> <span class="z-punctuation z-section z-block z-end z-java">}</span></span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> <span class="z-punctuation z-section z-block z-end z-java">}</span></span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Update last location </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-support z-class z-java">LastLocation</span> newLoc <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-meta z-instantiation z-java"><span class="z-keyword z-other z-storage z-new z-java">new</span> <span class="z-support z-class z-java">LastLocation</span><span class="z-meta z-parens z-constructor-arguments z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> newLoc<span class="z-punctuation z-accessor z-dot z-java">.</span>latitude <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> txn<span class="z-punctuation z-accessor z-dot z-java">.</span>latitude</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> newLoc<span class="z-punctuation z-accessor z-dot z-java">.</span>longitude <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> txn<span class="z-punctuation z-accessor z-dot z-java">.</span>longitude</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> newLoc<span class="z-punctuation z-accessor z-dot z-java">.</span>timestamp <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> txn<span class="z-punctuation z-accessor z-dot z-java">.</span>timestamp</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> lastLocationState<span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">update</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span>newLoc<span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-punctuation z-section z-block z-end z-java">}</span></span></span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-storage z-modifier z-java">private</span> <span class="z-storage z-type z-primitive z-java">double</span> <span class="z-meta z-method z-java"><span class="z-meta z-method z-identifier z-java"><span class="z-entity z-name z-function z-java">calculateDistance</span></span><span class="z-meta z-method z-parameters z-java"><span class="z-meta z-parens z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-storage z-type z-primitive z-java">double</span> <span class="z-variable z-parameter z-java">lat1</span><span class="z-punctuation z-separator z-comma z-java">,</span> <span class="z-storage z-type z-primitive z-java">double</span> <span class="z-variable z-parameter z-java">lon1</span><span class="z-punctuation z-separator z-comma z-java">,</span> <span class="z-storage z-type z-primitive z-java">double</span> <span class="z-variable z-parameter z-java">lat2</span><span class="z-punctuation z-separator z-comma z-java">,</span> <span class="z-storage z-type z-primitive z-java">double</span> <span class="z-variable z-parameter z-java">lon2</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span> <span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-punctuation z-section z-block z-begin z-java">{</span></span></span></span><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Haversine formula implementation </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-storage z-type z-primitive z-java">double</span> <span class="z-support z-class z-java">R</span> <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-constant z-numeric z-integer z-decimal z-java">6371</span></span><span class="z-punctuation z-terminator z-java">;</span> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Earth radius in km </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-storage z-type z-primitive z-java">double</span> dLat <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-support z-class z-java">Math</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">toRadians</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span>lat2 <span class="z-keyword z-operator z-arithmetic z-java">-</span> lat1<span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-storage z-type z-primitive z-java">double</span> dLon <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-support z-class z-java">Math</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">toRadians</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span>lon2 <span class="z-keyword z-operator z-arithmetic z-java">-</span> lon1<span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-storage z-type z-primitive z-java">double</span> a <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-support z-class z-java">Math</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">sin</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span>dLat<span class="z-keyword z-operator z-arithmetic z-java">/</span><span class="z-constant z-numeric z-integer z-decimal z-java">2</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span> <span class="z-keyword z-operator z-arithmetic z-java">*</span> <span class="z-support z-class z-java">Math</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">sin</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span>dLat<span class="z-keyword z-operator z-arithmetic z-java">/</span><span class="z-constant z-numeric z-integer z-decimal z-java">2</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span> <span class="z-keyword z-operator z-arithmetic z-java">+</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-assignment z-rhs z-java"> <span class="z-support z-class z-java">Math</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">cos</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-support z-class z-java">Math</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">toRadians</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span>lat1<span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span> <span class="z-keyword z-operator z-arithmetic z-java">*</span> <span class="z-support z-class z-java">Math</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">cos</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-support z-class z-java">Math</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">toRadians</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span>lat2<span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span> <span class="z-keyword z-operator z-arithmetic z-java">*</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-assignment z-rhs z-java"> <span class="z-support z-class z-java">Math</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">sin</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span>dLon<span class="z-keyword z-operator z-arithmetic z-java">/</span><span class="z-constant z-numeric z-integer z-decimal z-java">2</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span> <span class="z-keyword z-operator z-arithmetic z-java">*</span> <span class="z-support z-class z-java">Math</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">sin</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span>dLon<span class="z-keyword z-operator z-arithmetic z-java">/</span><span class="z-constant z-numeric z-integer z-decimal z-java">2</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-storage z-type z-primitive z-java">double</span> c <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-constant z-numeric z-integer z-decimal z-java">2</span> <span class="z-keyword z-operator z-arithmetic z-java">*</span> <span class="z-support z-class z-java">Math</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">atan2</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-support z-class z-java">Math</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">sqrt</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span>a<span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-separator z-comma z-java">,</span> <span class="z-support z-class z-java">Math</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">sqrt</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-constant z-numeric z-integer z-decimal z-java">1</span><span class="z-keyword z-operator z-arithmetic z-java">-</span>a<span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-keyword z-control z-flow z-return z-java">return</span> <span class="z-support z-class z-java">R</span> <span class="z-keyword z-operator z-arithmetic z-java">*</span> c<span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-punctuation z-section z-block z-end z-java">}</span></span></span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-punctuation z-section z-block z-end z-java">}</span></span></span></span> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Suspicious Sequence Detector: Detects patterns like test transactions </span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-meta z-class z-java"><span class="z-storage z-modifier z-java">public</span> <span class="z-storage z-modifier z-java">static</span> <span class="z-meta z-class z-identifier z-java"><span class="z-storage z-type z-java">class</span> <span class="z-entity z-name z-class z-java">SuspiciousSequenceDetector</span></span> </span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"> <span class="z-meta z-class z-extends z-java"><span class="z-keyword z-declaration z-extends z-java">extends</span> <span class="z-entity z-other z-inherited-class z-java">KeyedProcessFunction</span><span class="z-meta z-generic z-java"><span class="z-punctuation z-definition z-generic z-begin z-java">&lt;</span><span class="z-support z-class z-java">String</span><span class="z-punctuation z-separator z-comma z-java">,</span> <span class="z-support z-class z-java">Transaction</span><span class="z-punctuation z-separator z-comma z-java">,</span> <span class="z-support z-class z-java">FraudAlert</span><span class="z-punctuation z-definition z-generic z-end z-java">&gt;</span></span> </span><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-punctuation z-section z-block z-begin z-java">{</span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-storage z-modifier z-java">private</span> <span class="z-storage z-modifier z-java">transient</span> <span class="z-support z-class z-java">ListState</span><span class="z-meta z-generic z-java"><span class="z-punctuation z-definition z-generic z-begin z-java">&lt;</span><span class="z-support z-class z-java">Transaction</span><span class="z-punctuation z-definition z-generic z-end z-java">&gt;</span></span> <span class="z-meta z-field z-java">recentTransactionsState</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-meta z-annotation z-java"><span class="z-punctuation z-definition z-annotation z-java">@</span></span><span class="z-meta z-annotation z-java"><span class="z-meta z-annotation z-identifier z-java"><span class="z-variable z-annotation z-java">Override</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-annotation z-java"><span class="z-meta z-annotation z-identifier z-java"> </span></span><span class="z-storage z-modifier z-java">public</span> <span class="z-meta z-method z-java"><span class="z-storage z-type z-void z-java">void</span> <span class="z-meta z-method z-identifier z-java"><span class="z-entity z-name z-function z-java">open</span></span><span class="z-meta z-method z-parameters z-java"><span class="z-meta z-parens z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-support z-class z-java">Configuration</span> <span class="z-variable z-parameter z-java">parameters</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span> <span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-punctuation z-section z-block z-begin z-java">{</span></span></span></span><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-support z-class z-java">ListStateDescriptor</span><span class="z-meta z-generic z-java"><span class="z-punctuation z-definition z-generic z-begin z-java">&lt;</span><span class="z-support z-class z-java">Transaction</span><span class="z-punctuation z-definition z-generic z-end z-java">&gt;</span></span> descriptor <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-assignment z-rhs z-java"> <span class="z-meta z-instantiation z-java"><span class="z-keyword z-other z-storage z-new z-java">new</span> <span class="z-support z-class z-java">ListStateDescriptor</span><span class="z-punctuation z-definition z-generic z-diamond z-java">&lt;&gt;</span><span class="z-meta z-parens z-constructor-arguments z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-string z-quoted z-double z-java"><span class="z-punctuation z-definition z-string z-begin z-java">&quot;</span>recent-transactions<span class="z-punctuation z-definition z-string z-end z-java">&quot;</span></span><span class="z-punctuation z-separator z-comma z-java">,</span> <span class="z-support z-class z-java">Transaction</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-variable z-language z-java">class</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> recentTransactionsState <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">getRuntimeContext</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">getListState</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span>descriptor<span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-punctuation z-section z-block z-end z-java">}</span></span></span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-meta z-annotation z-java"><span class="z-punctuation z-definition z-annotation z-java">@</span></span><span class="z-meta z-annotation z-java"><span class="z-meta z-annotation z-identifier z-java"><span class="z-variable z-annotation z-java">Override</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-annotation z-java"><span class="z-meta z-annotation z-identifier z-java"> </span></span><span class="z-storage z-modifier z-java">public</span> <span class="z-meta z-method z-java"><span class="z-storage z-type z-void z-java">void</span> <span class="z-meta z-method z-identifier z-java"><span class="z-entity z-name z-function z-java">processElement</span></span><span class="z-meta z-method z-parameters z-java"><span class="z-meta z-parens z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-support z-class z-java">Transaction</span> <span class="z-variable z-parameter z-java">txn</span><span class="z-punctuation z-separator z-comma z-java">,</span> <span class="z-support z-class z-java">Context</span> <span class="z-variable z-parameter z-java">ctx</span><span class="z-punctuation z-separator z-comma z-java">,</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-parameters z-java"><span class="z-meta z-parens z-java"> <span class="z-support z-class z-java">Collector</span><span class="z-meta z-generic z-java"><span class="z-punctuation z-definition z-generic z-begin z-java">&lt;</span><span class="z-support z-class z-java">FraudAlert</span><span class="z-punctuation z-definition z-generic z-end z-java">&gt;</span></span> <span class="z-variable z-parameter z-java">out</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span> <span class="z-meta z-method z-throws z-java"><span class="z-keyword z-declaration z-throws z-java">throws</span> <span class="z-support z-class z-java">Exception</span> </span><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-punctuation z-section z-block z-begin z-java">{</span></span></span></span><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-support z-class z-java">List</span><span class="z-meta z-generic z-java"><span class="z-punctuation z-definition z-generic z-begin z-java">&lt;</span><span class="z-support z-class z-java">Transaction</span><span class="z-punctuation z-definition z-generic z-end z-java">&gt;</span></span> recentTxns <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-meta z-instantiation z-java"><span class="z-keyword z-other z-storage z-new z-java">new</span> <span class="z-support z-class z-java">ArrayList</span><span class="z-punctuation z-definition z-generic z-diamond z-java">&lt;&gt;</span><span class="z-meta z-parens z-constructor-arguments z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> recentTransactionsState<span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">get</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">forEach</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span>recentTxns<span class="z-punctuation z-accessor z-double-colon z-java">::</span><span class="z-variable z-function z-reference z-java">add</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Pattern 1: Multiple small transactions followed by large transaction </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-keyword z-control z-conditional z-if z-java">if</span> <span class="z-meta z-parens z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span>recentTxns<span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">size</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span> <span class="z-keyword z-operator z-comparison z-java">&gt;=</span> <span class="z-constant z-numeric z-integer z-decimal z-java">3</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span> <span class="z-meta z-block z-java"><span class="z-punctuation z-section z-block z-begin z-java">{</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> <span class="z-storage z-type z-primitive z-java">int</span> smallTxnCount <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-constant z-numeric z-integer z-decimal z-java">0</span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> <span class="z-keyword z-control z-loop z-for z-java">for</span> <span class="z-meta z-parens z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-support z-class z-java">Transaction</span> t <span class="z-keyword z-operator z-ternary z-java">:</span> recentTxns<span class="z-punctuation z-section z-parens z-end z-java">)</span></span> <span class="z-meta z-block z-java"><span class="z-punctuation z-section z-block z-begin z-java">{</span> </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-block z-java"> <span class="z-keyword z-control z-conditional z-if z-java">if</span> <span class="z-meta z-parens z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span>t<span class="z-punctuation z-accessor z-dot z-java">.</span>amount <span class="z-keyword z-operator z-comparison z-java">&lt;</span> <span class="z-constant z-numeric z-float z-decimal z-java">5<span class="z-punctuation z-separator z-decimal z-java">.</span>0</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span> smallTxnCount<span class="z-keyword z-operator z-increment-decrement z-java">++</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-block z-java"> <span class="z-punctuation z-section z-block z-end z-java">}</span></span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> <span class="z-keyword z-control z-conditional z-if z-java">if</span> <span class="z-meta z-parens z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span>smallTxnCount <span class="z-keyword z-operator z-comparison z-java">&gt;=</span> <span class="z-constant z-numeric z-integer z-decimal z-java">3</span> <span class="z-keyword z-operator z-logical z-java">&amp;&amp;</span> txn<span class="z-punctuation z-accessor z-dot z-java">.</span>amount <span class="z-keyword z-operator z-comparison z-java">&gt;</span> <span class="z-constant z-numeric z-integer z-decimal z-java">500</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span> <span class="z-meta z-block z-java"><span class="z-punctuation z-section z-block z-begin z-java">{</span> </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-block z-java"> <span class="z-support z-class z-java">FraudAlert</span> alert <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-meta z-instantiation z-java"><span class="z-keyword z-other z-storage z-new z-java">new</span> <span class="z-support z-class z-java">FraudAlert</span><span class="z-meta z-parens z-constructor-arguments z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-block z-java"> alert<span class="z-punctuation z-accessor z-dot z-java">.</span>transactionId <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> txn<span class="z-punctuation z-accessor z-dot z-java">.</span>transactionId</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-block z-java"> alert<span class="z-punctuation z-accessor z-dot z-java">.</span>userId <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> txn<span class="z-punctuation z-accessor z-dot z-java">.</span>userId</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-block z-java"> alert<span class="z-punctuation z-accessor z-dot z-java">.</span>alertType <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-string z-quoted z-double z-java"><span class="z-punctuation z-definition z-string z-begin z-java">&quot;</span>SUSPICIOUS_SEQUENCE<span class="z-punctuation z-definition z-string z-end z-java">&quot;</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-block z-java"> alert<span class="z-punctuation z-accessor z-dot z-java">.</span>reason <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-string z-quoted z-double z-java"><span class="z-punctuation z-definition z-string z-begin z-java">&quot;</span>Multiple small test transactions followed by large amount<span class="z-punctuation z-definition z-string z-end z-java">&quot;</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-block z-java"> alert<span class="z-punctuation z-accessor z-dot z-java">.</span>riskScore <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-constant z-numeric z-float z-decimal z-java">0<span class="z-punctuation z-separator z-decimal z-java">.</span>85</span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-block z-java"> alert<span class="z-punctuation z-accessor z-dot z-java">.</span>timestamp <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-support z-class z-java">System</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">currentTimeMillis</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-block z-java"> out<span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">collect</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span>alert<span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-block z-java"> <span class="z-punctuation z-section z-block z-end z-java">}</span></span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> <span class="z-punctuation z-section z-block z-end z-java">}</span></span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Pattern 2: Transactions at suspicious merchant categories </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-support z-class z-java">String</span><span class="z-storage z-modifier z-array z-java">[]</span> suspiciousMerchants <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-meta z-block z-java"><span class="z-punctuation z-section z-block z-begin z-java">{</span><span class="z-string z-quoted z-double z-java"><span class="z-punctuation z-definition z-string z-begin z-java">&quot;</span>WIRE_TRANSFER<span class="z-punctuation z-definition z-string z-end z-java">&quot;</span></span><span class="z-punctuation z-separator z-comma z-java">,</span> <span class="z-string z-quoted z-double z-java"><span class="z-punctuation z-definition z-string z-begin z-java">&quot;</span>CRYPTO<span class="z-punctuation z-definition z-string z-end z-java">&quot;</span></span><span class="z-punctuation z-separator z-comma z-java">,</span> <span class="z-string z-quoted z-double z-java"><span class="z-punctuation z-definition z-string z-begin z-java">&quot;</span>GIFT_CARDS<span class="z-punctuation z-definition z-string z-end z-java">&quot;</span></span><span class="z-punctuation z-section z-block z-end z-java">}</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-keyword z-control z-conditional z-if z-java">if</span> <span class="z-meta z-parens z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-support z-class z-java">Arrays</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">asList</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span>suspiciousMerchants<span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">contains</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span>txn<span class="z-punctuation z-accessor z-dot z-java">.</span>merchantCategory<span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span> <span class="z-meta z-block z-java"><span class="z-punctuation z-section z-block z-begin z-java">{</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> <span class="z-support z-class z-java">FraudAlert</span> alert <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-meta z-instantiation z-java"><span class="z-keyword z-other z-storage z-new z-java">new</span> <span class="z-support z-class z-java">FraudAlert</span><span class="z-meta z-parens z-constructor-arguments z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> alert<span class="z-punctuation z-accessor z-dot z-java">.</span>transactionId <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> txn<span class="z-punctuation z-accessor z-dot z-java">.</span>transactionId</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> alert<span class="z-punctuation z-accessor z-dot z-java">.</span>userId <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> txn<span class="z-punctuation z-accessor z-dot z-java">.</span>userId</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> alert<span class="z-punctuation z-accessor z-dot z-java">.</span>alertType <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-string z-quoted z-double z-java"><span class="z-punctuation z-definition z-string z-begin z-java">&quot;</span>SUSPICIOUS_MERCHANT<span class="z-punctuation z-definition z-string z-end z-java">&quot;</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> alert<span class="z-punctuation z-accessor z-dot z-java">.</span>reason <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-string z-quoted z-double z-java"><span class="z-punctuation z-definition z-string z-begin z-java">&quot;</span>Transaction at high-risk merchant category: <span class="z-punctuation z-definition z-string z-end z-java">&quot;</span></span> <span class="z-keyword z-operator z-arithmetic z-java">+</span> txn<span class="z-punctuation z-accessor z-dot z-java">.</span>merchantCategory</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> alert<span class="z-punctuation z-accessor z-dot z-java">.</span>riskScore <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-constant z-numeric z-float z-decimal z-java">0<span class="z-punctuation z-separator z-decimal z-java">.</span>6</span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> alert<span class="z-punctuation z-accessor z-dot z-java">.</span>timestamp <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-support z-class z-java">System</span><span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">currentTimeMillis</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> out<span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">collect</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span>alert<span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> <span class="z-punctuation z-section z-block z-end z-java">}</span></span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Keep only last 5 transactions </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> recentTxns<span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">add</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span>txn<span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-keyword z-control z-conditional z-if z-java">if</span> <span class="z-meta z-parens z-java"><span class="z-punctuation z-section z-parens z-begin z-java">(</span>recentTxns<span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">size</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span> <span class="z-keyword z-operator z-comparison z-java">&gt;</span> <span class="z-constant z-numeric z-integer z-decimal z-java">5</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span> <span class="z-meta z-block z-java"><span class="z-punctuation z-section z-block z-begin z-java">{</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> recentTxns<span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">remove</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span><span class="z-constant z-numeric z-integer z-decimal z-java">0</span><span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"><span class="z-meta z-block z-java"> <span class="z-punctuation z-section z-block z-end z-java">}</span></span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> recentTransactionsState<span class="z-punctuation z-accessor z-dot z-java">.</span><span class="z-meta z-function-call z-java"><span class="z-variable z-function z-java">update</span><span class="z-punctuation z-section z-parens z-begin z-java">(</span>recentTxns<span class="z-punctuation z-section z-parens z-end z-java">)</span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-method z-java"><span class="z-meta z-method z-body z-java"> <span class="z-punctuation z-section z-block z-end z-java">}</span></span></span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-punctuation z-section z-block z-end z-java">}</span></span></span></span> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-comment z-line z-double-slash z-java"><span class="z-punctuation z-definition z-comment z-java">//</span> Helper classes </span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-meta z-class z-java"><span class="z-storage z-modifier z-java">public</span> <span class="z-storage z-modifier z-java">static</span> <span class="z-meta z-class z-identifier z-java"><span class="z-storage z-type z-java">class</span> <span class="z-entity z-name z-class z-java">TransactionStats</span></span> <span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-punctuation z-section z-block z-begin z-java">{</span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-storage z-modifier z-java">public</span> <span class="z-storage z-type z-primitive z-java">int</span> <span class="z-meta z-field z-java">count</span> <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-constant z-numeric z-integer z-decimal z-java">0</span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-storage z-modifier z-java">public</span> <span class="z-storage z-type z-primitive z-java">double</span> <span class="z-meta z-field z-java">totalAmount</span> <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-constant z-numeric z-float z-decimal z-java">0<span class="z-punctuation z-separator z-decimal z-java">.</span>0</span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-storage z-modifier z-java">public</span> <span class="z-storage z-type z-primitive z-java">double</span> <span class="z-meta z-field z-java">sumOfSquares</span> <span class="z-meta z-assignment z-rhs z-java"><span class="z-keyword z-operator z-assignment z-java">=</span> <span class="z-constant z-numeric z-float z-decimal z-java">0<span class="z-punctuation z-separator z-decimal z-java">.</span>0</span></span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-punctuation z-section z-block z-end z-java">}</span></span></span></span> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-meta z-class z-java"><span class="z-storage z-modifier z-java">public</span> <span class="z-storage z-modifier z-java">static</span> <span class="z-meta z-class z-identifier z-java"><span class="z-storage z-type z-java">class</span> <span class="z-entity z-name z-class z-java">LastLocation</span></span> <span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-punctuation z-section z-block z-begin z-java">{</span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-storage z-modifier z-java">public</span> <span class="z-storage z-type z-primitive z-java">double</span> <span class="z-meta z-field z-java">latitude</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-storage z-modifier z-java">public</span> <span class="z-storage z-type z-primitive z-java">double</span> <span class="z-meta z-field z-java">longitude</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-storage z-modifier z-java">public</span> <span class="z-storage z-type z-primitive z-java">long</span> <span class="z-meta z-field z-java">timestamp</span><span class="z-punctuation z-terminator z-java">;</span> </span></span></span></span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"> <span class="z-punctuation z-section z-block z-end z-java">}</span></span></span></span> </span></span></span></span><span class="z-source z-java"><span class="z-meta z-class z-java"><span class="z-meta z-class z-body z-java"><span class="z-meta z-block z-java"><span class="z-punctuation z-section z-block z-end z-java">}</span></span></span></span> </span></code></pre> <p><strong>Why Flink for this use case?</strong></p> <ul> <li><strong>True real-time processing</strong>: Detects fraud within milliseconds</li> <li><strong>Stateful computations</strong>: Maintains user transaction history in memory</li> <li><strong>Event time processing</strong>: Handles late-arriving transactions correctly</li> <li><strong>Exactly-once guarantees</strong>: Ensures no alert is missed or duplicated</li> <li><strong>Complex event processing</strong>: Detects sophisticated fraud patterns across multiple transactions</li> </ul> <h2 id="apache-airflow-the-workflow-orchestrator">Apache Airflow: The Workflow Orchestrator<a class="zola-anchor" href="#apache-airflow-the-workflow-orchestrator" aria-label="Anchor link for: apache-airflow-the-workflow-orchestrator" style="visibility: hidden;"></a> </h2> <h3 id="what-is-apache-airflow">What is Apache Airflow?<a class="zola-anchor" href="#what-is-apache-airflow" aria-label="Anchor link for: what-is-apache-airflow" style="visibility: hidden;"></a> </h3> <p>Apache Airflow is a platform to programmatically author, schedule, and monitor workflows. It's not a data processing engine itself but rather orchestrates data pipelines that may use Spark, Flink, or other tools.</p> <h3 id="key-features-2">Key Features<a class="zola-anchor" href="#key-features-2" aria-label="Anchor link for: key-features-2" style="visibility: hidden;"></a> </h3> <ul> <li><strong>DAG-based workflows</strong>: Define workflows as Directed Acyclic Graphs</li> <li><strong>Extensible</strong>: Rich library of operators for various systems</li> <li><strong>Dynamic pipelines</strong>: Generate pipelines dynamically using Python</li> <li><strong>Rich UI</strong>: Web interface to monitor and manage workflows</li> <li><strong>Scheduling</strong>: Cron-based scheduling with dependency management</li> <li><strong>Task retries and alerts</strong>: Automatic retry and notification on failures</li> </ul> <h3 id="real-world-example-end-to-end-ml-pipeline-orchestration">Real-World Example: End-to-End ML Pipeline Orchestration<a class="zola-anchor" href="#real-world-example-end-to-end-ml-pipeline-orchestration" aria-label="Anchor link for: real-world-example-end-to-end-ml-pipeline-orchestration" style="visibility: hidden;"></a> </h3> <p>Let's build a complete ML pipeline that orchestrates data extraction, transformation, model training, and deployment.</p> <p><strong>Scenario</strong>: Build a daily ML pipeline that:</p> <ol> <li>Extracts data from multiple sources</li> <li>Runs data quality checks</li> <li>Processes data with Spark</li> <li>Trains a machine learning model</li> <li>Validates model performance</li> <li>Deploys model if metrics are acceptable</li> <li>Sends notifications on success/failure</li> </ol> <pre data-lang="python" class="language-python z-code"><code class="language-python" data-lang="python"><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-from z-python">from</span></span><span class="z-meta z-statement z-import z-python"><span class="z-meta z-import-source z-python"> <span class="z-meta z-import-path z-python"><span class="z-meta z-import-name z-python">datetime</span></span> <span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span></span></span></span><span class="z-meta z-statement z-import z-python"></span><span class="z-meta z-statement z-import z-python"> <span class="z-meta z-generic-name z-python">datetime</span><span class="z-punctuation z-separator z-import-list z-python">,</span> <span class="z-meta z-generic-name z-python">timedelta</span></span> </span><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-from z-python">from</span></span><span class="z-meta z-statement z-import z-python"><span class="z-meta z-import-source z-python"> <span class="z-meta z-import-path z-python"><span class="z-meta z-import-name z-python">airflow</span></span> <span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span></span></span></span><span class="z-meta z-statement z-import z-python"></span><span class="z-meta z-statement z-import z-python"> <span class="z-variable z-other z-constant z-python">DAG</span></span> </span><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-from z-python">from</span></span><span class="z-meta z-statement z-import z-python"><span class="z-meta z-import-source z-python"> <span class="z-meta z-import-path z-python"><span class="z-meta z-import-name z-python">airflow</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">operators</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">python</span></span> <span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span></span></span></span><span class="z-meta z-statement z-import z-python"></span><span class="z-meta z-statement z-import z-python"> <span class="z-meta z-generic-name z-python">PythonOperator</span><span class="z-punctuation z-separator z-import-list z-python">,</span> <span class="z-meta z-generic-name z-python">BranchPythonOperator</span></span> </span><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-from z-python">from</span></span><span class="z-meta z-statement z-import z-python"><span class="z-meta z-import-source z-python"> <span class="z-meta z-import-path z-python"><span class="z-meta z-import-name z-python">airflow</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">operators</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">bash</span></span> <span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span></span></span></span><span class="z-meta z-statement z-import z-python"></span><span class="z-meta z-statement z-import z-python"> <span class="z-meta z-generic-name z-python">BashOperator</span></span> </span><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-from z-python">from</span></span><span class="z-meta z-statement z-import z-python"><span class="z-meta z-import-source z-python"> <span class="z-meta z-import-path z-python"><span class="z-meta z-import-name z-python">airflow</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">operators</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">dummy</span></span> <span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span></span></span></span><span class="z-meta z-statement z-import z-python"></span><span class="z-meta z-statement z-import z-python"> <span class="z-meta z-generic-name z-python">DummyOperator</span></span> </span><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-from z-python">from</span></span><span class="z-meta z-statement z-import z-python"><span class="z-meta z-import-source z-python"> <span class="z-meta z-import-path z-python"><span class="z-meta z-import-name z-python">airflow</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">providers</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">amazon</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">aws</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">operators</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">s3</span></span> <span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span></span></span></span><span class="z-meta z-statement z-import z-python"></span><span class="z-meta z-statement z-import z-python"> <span class="z-meta z-generic-name z-python">S3CreateBucketOperator</span></span> </span><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-from z-python">from</span></span><span class="z-meta z-statement z-import z-python"><span class="z-meta z-import-source z-python"> <span class="z-meta z-import-path z-python"><span class="z-meta z-import-name z-python">airflow</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">providers</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">amazon</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">aws</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">operators</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">emr</span></span> <span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span></span></span></span><span class="z-meta z-statement z-import z-python"></span><span class="z-meta z-statement z-import z-python"> <span class="z-meta z-generic-name z-python">EmrCreateJobFlowOperator</span><span class="z-punctuation z-separator z-import-list z-python">,</span> <span class="z-meta z-generic-name z-python">EmrTerminateJobFlowOperator</span></span> </span><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-from z-python">from</span></span><span class="z-meta z-statement z-import z-python"><span class="z-meta z-import-source z-python"> <span class="z-meta z-import-path z-python"><span class="z-meta z-import-name z-python">airflow</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">providers</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">amazon</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">aws</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">sensors</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">emr</span></span> <span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span></span></span></span><span class="z-meta z-statement z-import z-python"></span><span class="z-meta z-statement z-import z-python"> <span class="z-meta z-generic-name z-python">EmrJobFlowSensor</span></span> </span><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-from z-python">from</span></span><span class="z-meta z-statement z-import z-python"><span class="z-meta z-import-source z-python"> <span class="z-meta z-import-path z-python"><span class="z-meta z-import-name z-python">airflow</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">providers</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">postgres</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">operators</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">postgres</span></span> <span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span></span></span></span><span class="z-meta z-statement z-import z-python"></span><span class="z-meta z-statement z-import z-python"> <span class="z-meta z-generic-name z-python">PostgresOperator</span></span> </span><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-from z-python">from</span></span><span class="z-meta z-statement z-import z-python"><span class="z-meta z-import-source z-python"> <span class="z-meta z-import-path z-python"><span class="z-meta z-import-name z-python">airflow</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">providers</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">http</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">operators</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">http</span></span> <span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span></span></span></span><span class="z-meta z-statement z-import z-python"></span><span class="z-meta z-statement z-import z-python"> <span class="z-meta z-generic-name z-python">SimpleHttpOperator</span></span> </span><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-from z-python">from</span></span><span class="z-meta z-statement z-import z-python"><span class="z-meta z-import-source z-python"> <span class="z-meta z-import-path z-python"><span class="z-meta z-import-name z-python">airflow</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">providers</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">slack</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">operators</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">slack_webhook</span></span> <span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span></span></span></span><span class="z-meta z-statement z-import z-python"></span><span class="z-meta z-statement z-import z-python"> <span class="z-meta z-generic-name z-python">SlackWebhookOperator</span></span> </span><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-from z-python">from</span></span><span class="z-meta z-statement z-import z-python"><span class="z-meta z-import-source z-python"> <span class="z-meta z-import-path z-python"><span class="z-meta z-import-name z-python">airflow</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">models</span></span> <span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span></span></span></span><span class="z-meta z-statement z-import z-python"></span><span class="z-meta z-statement z-import z-python"> <span class="z-meta z-generic-name z-python">Variable</span></span> </span><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-from z-python">from</span></span><span class="z-meta z-statement z-import z-python"><span class="z-meta z-import-source z-python"> <span class="z-meta z-import-path z-python"><span class="z-meta z-import-name z-python">airflow</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">utils</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">trigger_rule</span></span> <span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span></span></span></span><span class="z-meta z-statement z-import z-python"></span><span class="z-meta z-statement z-import z-python"> <span class="z-meta z-generic-name z-python">TriggerRule</span></span> </span><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">json</span></span></span> </span><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">boto3</span></span></span> </span><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">pandas</span></span> <span class="z-keyword z-control z-import z-as z-python">as</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">pd</span></span></span> </span><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">numpy</span></span> <span class="z-keyword z-control z-import z-as z-python">as</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">np</span></span></span> </span><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-from z-python">from</span></span><span class="z-meta z-statement z-import z-python"><span class="z-meta z-import-source z-python"> <span class="z-meta z-import-path z-python"><span class="z-meta z-import-name z-python">sklearn</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">ensemble</span></span> <span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span></span></span></span><span class="z-meta z-statement z-import z-python"></span><span class="z-meta z-statement z-import z-python"> <span class="z-meta z-generic-name z-python">RandomForestClassifier</span></span> </span><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-from z-python">from</span></span><span class="z-meta z-statement z-import z-python"><span class="z-meta z-import-source z-python"> <span class="z-meta z-import-path z-python"><span class="z-meta z-import-name z-python">sklearn</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-meta z-import-name z-python">metrics</span></span> <span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span></span></span></span><span class="z-meta z-statement z-import z-python"></span><span class="z-meta z-statement z-import z-python"> <span class="z-meta z-generic-name z-python">accuracy_score</span><span class="z-punctuation z-separator z-import-list z-python">,</span> <span class="z-meta z-generic-name z-python">precision_score</span><span class="z-punctuation z-separator z-import-list z-python">,</span> <span class="z-meta z-generic-name z-python">recall_score</span><span class="z-punctuation z-separator z-import-list z-python">,</span> <span class="z-meta z-generic-name z-python">f1_score</span></span> </span><span class="z-source z-python"><span class="z-meta z-statement z-import z-python"><span class="z-keyword z-control z-import z-python">import</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">joblib</span></span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Default arguments for the DAG </span></span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">default_args</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-mapping-or-set z-python"><span class="z-punctuation z-section z-mapping-or-set z-begin z-python">{</span> </span></span><span class="z-source z-python"><span class="z-meta z-mapping-or-set z-python"></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">owner<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">data-science-team<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">depends_on_past<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-constant z-language z-python">False</span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">start_date<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">datetime</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-constant z-numeric z-integer z-decimal z-python">2026</span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-constant z-numeric z-integer z-decimal z-python">1</span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-constant z-numeric z-integer z-decimal z-python">1</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">email<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-punctuation z-section z-sequence z-begin z-python">[</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">datascience@company.com<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-section z-sequence z-end z-python">]</span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">email_on_failure<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-constant z-language z-python">True</span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">email_on_retry<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-constant z-language z-python">False</span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">retries<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-constant z-numeric z-integer z-decimal z-python">2</span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">retry_delay<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">timedelta</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-variable z-parameter z-python">minutes</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-constant z-numeric z-integer z-decimal z-python">5</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">execution_timeout<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">timedelta</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-variable z-parameter z-python">hours</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-constant z-numeric z-integer z-decimal z-python">4</span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-mapping z-python"><span class="z-punctuation z-section z-mapping z-end z-python">}</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Create the DAG </span></span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">dag</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">DAG</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">ml_pipeline_customer_churn<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">default_args</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">default_args</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">description</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">End-to-end ML pipeline for customer churn prediction<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">schedule_interval</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">0 2 * * *<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Run daily at 2 AM </span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">catchup</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-constant z-language z-python">False</span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">max_active_runs</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-constant z-numeric z-integer z-decimal z-python">1</span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">tags</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-sequence z-list z-python"><span class="z-punctuation z-section z-sequence z-begin z-python">[</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">ml<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">production<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">churn<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-section z-sequence z-end z-python">]</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Task 1: Check data availability in source systems </span></span><span class="z-source z-python"><span class="z-meta z-function z-python"><span class="z-storage z-type z-function z-python"><span class="z-keyword z-declaration z-function z-python">def</span></span> <span class="z-entity z-name z-function z-python"><span class="z-meta z-generic-name z-python">check_data_availability</span></span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-begin z-python">(</span></span><span class="z-meta z-function z-parameters z-python"><span class="z-keyword z-operator z-unpacking z-mapping z-python">**</span><span class="z-variable z-parameter z-python">context</span><span class="z-punctuation z-section z-parameters z-end z-python">)</span></span><span class="z-meta z-function z-python"><span class="z-punctuation z-section z-function z-begin z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-comment z-block z-documentation z-python"><span class="z-punctuation z-definition z-comment z-begin z-python">&quot;&quot;&quot;</span>Check if all required data sources have data for execution date<span class="z-punctuation z-definition z-comment z-end z-python">&quot;&quot;&quot;</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">execution_date</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-item-access z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">context</span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-begin z-python">[</span></span><span class="z-meta z-item-access z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">execution_date<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-end z-python">]</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">date_str</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">execution_date</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">strftime</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-constant z-other z-placeholder z-python">%Y</span>-<span class="z-constant z-other z-placeholder z-python">%m</span>-<span class="z-constant z-other z-placeholder z-python">%d</span><span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">s3</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">boto3</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">client</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">s3<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">bucket</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">company-data-lake<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Check multiple data sources </span></span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">sources</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-sequence z-list z-python"><span class="z-punctuation z-section z-sequence z-begin z-python">[</span> </span></span><span class="z-source z-python"><span class="z-meta z-sequence z-list z-python"> <span class="z-storage z-type z-string z-python">f</span><span class="z-meta z-string z-interpolated z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-interpolated z-python"><span class="z-string z-quoted z-single z-python">raw/customer_data/</span><span class="z-meta z-interpolation z-python"><span class="z-punctuation z-section z-interpolation z-begin z-python">{</span><span class="z-source z-python z-embedded"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">date_str</span></span></span></span><span class="z-meta z-interpolation z-python"><span class="z-punctuation z-section z-interpolation z-end z-python">}</span></span><span class="z-string z-quoted z-single z-python">/<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-sequence z-list z-python"> <span class="z-storage z-type z-string z-python">f</span><span class="z-meta z-string z-interpolated z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-interpolated z-python"><span class="z-string z-quoted z-single z-python">raw/transactions/</span><span class="z-meta z-interpolation z-python"><span class="z-punctuation z-section z-interpolation z-begin z-python">{</span><span class="z-source z-python z-embedded"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">date_str</span></span></span></span><span class="z-meta z-interpolation z-python"><span class="z-punctuation z-section z-interpolation z-end z-python">}</span></span><span class="z-string z-quoted z-single z-python">/<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-sequence z-list z-python"> <span class="z-storage z-type z-string z-python">f</span><span class="z-meta z-string z-interpolated z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-interpolated z-python"><span class="z-string z-quoted z-single z-python">raw/support_tickets/</span><span class="z-meta z-interpolation z-python"><span class="z-punctuation z-section z-interpolation z-begin z-python">{</span><span class="z-source z-python z-embedded"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">date_str</span></span></span></span><span class="z-meta z-interpolation z-python"><span class="z-punctuation z-section z-interpolation z-end z-python">}</span></span><span class="z-string z-quoted z-single z-python">/<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-sequence z-list z-python"> <span class="z-storage z-type z-string z-python">f</span><span class="z-meta z-string z-interpolated z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-interpolated z-python"><span class="z-string z-quoted z-single z-python">raw/product_usage/</span><span class="z-meta z-interpolation z-python"><span class="z-punctuation z-section z-interpolation z-begin z-python">{</span><span class="z-source z-python z-embedded"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">date_str</span></span></span></span><span class="z-meta z-interpolation z-python"><span class="z-punctuation z-section z-interpolation z-end z-python">}</span></span><span class="z-string z-quoted z-single z-python">/<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span> </span></span><span class="z-source z-python"><span class="z-meta z-sequence z-list z-python"> <span class="z-punctuation z-section z-sequence z-end z-python">]</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">missing_sources</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-sequence z-list z-empty z-python"><span class="z-punctuation z-section z-sequence z-begin z-python">[</span><span class="z-punctuation z-section z-sequence z-end z-python">]</span></span> </span><span class="z-source z-python"> <span class="z-meta z-statement z-loop z-for z-python"><span class="z-keyword z-control z-loop z-for z-python">for</span> <span class="z-meta z-generic-name z-python">source</span> <span class="z-keyword z-control z-loop z-for z-in z-python">in</span></span><span class="z-meta z-statement z-loop z-for z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">sources</span></span></span><span class="z-meta z-statement z-loop z-for z-python"><span class="z-punctuation z-section z-block z-loop z-for z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-statement z-exception z-try z-python"><span class="z-keyword z-control z-exception z-try z-python">try</span><span class="z-punctuation z-section z-block z-exception z-try z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">response</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">s3</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">list_objects_v2</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-variable z-parameter z-python">Bucket</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">bucket</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-variable z-parameter z-python">Prefix</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">source</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-meta z-statement z-conditional z-if z-python"><span class="z-keyword z-control z-conditional z-if z-python">if</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">Contents<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span> <span class="z-keyword z-operator z-logical z-python">not</span> <span class="z-keyword z-operator z-logical z-python">in</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">response</span></span> <span class="z-keyword z-operator z-logical z-python">or</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">len</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-item-access z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">response</span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-begin z-python">[</span></span><span class="z-meta z-item-access z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">Contents<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-end z-python">]</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> <span class="z-keyword z-operator z-comparison z-python">==</span> <span class="z-constant z-numeric z-integer z-decimal z-python">0</span><span class="z-punctuation z-section z-block z-conditional z-if z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">missing_sources</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">append</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">source</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-meta z-statement z-exception z-catch z-python"><span class="z-keyword z-control z-exception z-catch z-python">except</span> <span class="z-support z-type z-exception z-python">Exception</span> <span class="z-keyword z-control z-exception z-catch z-as z-python">as</span></span><span class="z-meta z-statement z-exception z-catch z-python"> <span class="z-meta z-generic-name z-python">e</span></span><span class="z-meta z-statement z-exception z-catch z-python"><span class="z-punctuation z-section z-block z-exception z-catch z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">missing_sources</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">append</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">source</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-meta z-statement z-conditional z-if z-python"><span class="z-keyword z-control z-conditional z-if z-python">if</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">missing_sources</span></span><span class="z-punctuation z-section z-block z-conditional z-if z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-statement z-raise z-python"><span class="z-keyword z-control z-flow z-raise z-python">raise</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-type z-exception z-python">ValueError</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-storage z-type z-string z-python">f</span><span class="z-meta z-string z-interpolated z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-interpolated z-python"><span class="z-string z-quoted z-double z-python">Missing data sources: </span><span class="z-meta z-interpolation z-python"><span class="z-punctuation z-section z-interpolation z-begin z-python">{</span><span class="z-source z-python z-embedded"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">missing_sources</span></span></span></span><span class="z-meta z-interpolation z-python"><span class="z-punctuation z-section z-interpolation z-end z-python">}</span></span><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">print</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-storage z-type z-string z-python">f</span><span class="z-meta z-string z-interpolated z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-interpolated z-python"><span class="z-string z-quoted z-double z-python">All data sources available for </span><span class="z-meta z-interpolation z-python"><span class="z-punctuation z-section z-interpolation z-begin z-python">{</span><span class="z-source z-python z-embedded"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">date_str</span></span></span></span><span class="z-meta z-interpolation z-python"><span class="z-punctuation z-section z-interpolation z-end z-python">}</span></span><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-keyword z-control z-flow z-return z-python">return</span> <span class="z-constant z-language z-python">True</span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">check_data_task</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">PythonOperator</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">task_id</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">check_data_availability<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">python_callable</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">check_data_availability</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">provide_context</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-constant z-language z-python">True</span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">dag</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">dag</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Task 2: Run data quality checks </span></span><span class="z-source z-python"><span class="z-meta z-function z-python"><span class="z-storage z-type z-function z-python"><span class="z-keyword z-declaration z-function z-python">def</span></span> <span class="z-entity z-name z-function z-python"><span class="z-meta z-generic-name z-python">run_data_quality_checks</span></span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-begin z-python">(</span></span><span class="z-meta z-function z-parameters z-python"><span class="z-keyword z-operator z-unpacking z-mapping z-python">**</span><span class="z-variable z-parameter z-python">context</span><span class="z-punctuation z-section z-parameters z-end z-python">)</span></span><span class="z-meta z-function z-python"><span class="z-punctuation z-section z-function z-begin z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-comment z-block z-documentation z-python"><span class="z-punctuation z-definition z-comment z-begin z-python">&quot;&quot;&quot;</span>Run data quality checks on raw data<span class="z-punctuation z-definition z-comment z-end z-python">&quot;&quot;&quot;</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">execution_date</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-item-access z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">context</span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-begin z-python">[</span></span><span class="z-meta z-item-access z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">execution_date<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-end z-python">]</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">date_str</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">execution_date</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">strftime</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-constant z-other z-placeholder z-python">%Y</span>-<span class="z-constant z-other z-placeholder z-python">%m</span>-<span class="z-constant z-other z-placeholder z-python">%d</span><span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Read sample of data from S3 </span></span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">s3</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">boto3</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">client</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">s3<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">quality_issues</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-sequence z-list z-empty z-python"><span class="z-punctuation z-section z-sequence z-begin z-python">[</span><span class="z-punctuation z-section z-sequence z-end z-python">]</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Check 1: Null value percentage </span></span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> (In production, you&#39;d read actual data) </span></span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">null_percentage</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-constant z-numeric z-float z-decimal z-python">0<span class="z-punctuation z-separator z-decimal z-python">.</span>05</span> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> 5% nulls </span></span><span class="z-source z-python"> <span class="z-meta z-statement z-conditional z-if z-python"><span class="z-keyword z-control z-conditional z-if z-python">if</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">null_percentage</span></span> <span class="z-keyword z-operator z-comparison z-python">&gt;</span> <span class="z-constant z-numeric z-float z-decimal z-python">0<span class="z-punctuation z-separator z-decimal z-python">.</span>10</span><span class="z-punctuation z-section z-block z-conditional z-if z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">quality_issues</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">append</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-storage z-type z-string z-python">f</span><span class="z-meta z-string z-interpolated z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-interpolated z-python"><span class="z-string z-quoted z-double z-python">High null percentage: </span><span class="z-meta z-interpolation z-python"><span class="z-punctuation z-section z-interpolation z-begin z-python">{</span><span class="z-source z-python z-embedded"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">null_percentage</span></span><span class="z-keyword z-operator z-arithmetic z-python">*</span><span class="z-constant z-numeric z-integer z-decimal z-python">100</span></span></span><span class="z-meta z-interpolation z-python"><span class="z-punctuation z-section z-interpolation z-end z-python">}</span></span><span class="z-string z-quoted z-double z-python">%<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Check 2: Data freshness </span></span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Ensure data is not older than expected </span></span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Check 3: Schema validation </span></span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Validate that all expected columns exist </span></span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Check 4: Value ranges </span></span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Check that numeric values are within expected ranges </span></span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-meta z-statement z-conditional z-if z-python"><span class="z-keyword z-control z-conditional z-if z-python">if</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">quality_issues</span></span><span class="z-punctuation z-section z-block z-conditional z-if z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-meta z-statement z-raise z-python"><span class="z-keyword z-control z-flow z-raise z-python">raise</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-type z-exception z-python">ValueError</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-storage z-type z-string z-python">f</span><span class="z-meta z-string z-interpolated z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-interpolated z-python"><span class="z-string z-quoted z-double z-python">Data quality issues found: </span><span class="z-meta z-interpolation z-python"><span class="z-punctuation z-section z-interpolation z-begin z-python">{</span><span class="z-source z-python z-embedded"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">quality_issues</span></span></span></span><span class="z-meta z-interpolation z-python"><span class="z-punctuation z-section z-interpolation z-end z-python">}</span></span><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">print</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-storage z-type z-string z-python">f</span><span class="z-meta z-string z-interpolated z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-interpolated z-python"><span class="z-string z-quoted z-double z-python">Data quality checks passed for </span><span class="z-meta z-interpolation z-python"><span class="z-punctuation z-section z-interpolation z-begin z-python">{</span><span class="z-source z-python z-embedded"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">date_str</span></span></span></span><span class="z-meta z-interpolation z-python"><span class="z-punctuation z-section z-interpolation z-end z-python">}</span></span><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-keyword z-control z-flow z-return z-python">return</span> <span class="z-constant z-language z-python">True</span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">data_quality_task</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">PythonOperator</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">task_id</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">data_quality_checks<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">python_callable</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">run_data_quality_checks</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">provide_context</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-constant z-language z-python">True</span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">dag</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">dag</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Task 3: Submit Spark job to EMR for data processing </span></span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-other z-constant z-python">EMR_CONFIG</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-mapping-or-set z-python"><span class="z-punctuation z-section z-mapping-or-set z-begin z-python">{</span> </span></span><span class="z-source z-python"><span class="z-meta z-mapping-or-set z-python"></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">Name<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">ChurnDataProcessing<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">ReleaseLabel<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">emr-6.10.0<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">Applications<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-punctuation z-section z-sequence z-begin z-python">[</span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-section z-mapping z-begin z-python">{</span></span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">Name<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">Spark<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-section z-mapping z-end z-python">}</span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> <span class="z-meta z-mapping z-python"><span class="z-punctuation z-section z-mapping z-begin z-python">{</span></span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">Name<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">Hadoop<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-section z-mapping z-end z-python">}</span></span><span class="z-punctuation z-section z-sequence z-end z-python">]</span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">Instances<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-mapping-or-set z-python"><span class="z-punctuation z-section z-mapping-or-set z-begin z-python">{</span> </span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-mapping-or-set z-python"></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">InstanceGroups<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-punctuation z-section z-sequence z-begin z-python">[</span> </span></span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"> <span class="z-meta z-mapping-or-set z-python"><span class="z-punctuation z-section z-mapping-or-set z-begin z-python">{</span> </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-meta z-mapping-or-set z-python"></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">Name<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">Master<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">Market<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">ON_DEMAND<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">InstanceRole<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">MASTER<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">InstanceType<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">r5.xlarge<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">InstanceCount<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-constant z-numeric z-integer z-decimal z-python">1</span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-meta z-mapping z-python"> <span class="z-punctuation z-section z-mapping z-end z-python">}</span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> </span></span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"> <span class="z-meta z-mapping-or-set z-python"><span class="z-punctuation z-section z-mapping-or-set z-begin z-python">{</span> </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-meta z-mapping-or-set z-python"></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">Name<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">Worker<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">Market<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">SPOT<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">InstanceRole<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">CORE<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">InstanceType<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">r5.2xlarge<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">InstanceCount<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-constant z-numeric z-integer z-decimal z-python">3</span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-meta z-mapping z-python"> <span class="z-punctuation z-section z-mapping z-end z-python">}</span></span> </span></span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"> <span class="z-punctuation z-section z-sequence z-end z-python">]</span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">KeepJobFlowAliveWhenNoSteps<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-constant z-language z-python">True</span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">TerminationProtected<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-constant z-language z-python">False</span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-mapping z-python"> <span class="z-punctuation z-section z-mapping z-end z-python">}</span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">JobFlowRole<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">EMR_EC2_DefaultRole<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">ServiceRole<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">EMR_DefaultRole<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">LogUri<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">s3://company-emr-logs/<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">Steps<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-punctuation z-section z-sequence z-begin z-python">[</span> </span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"> <span class="z-meta z-mapping-or-set z-python"><span class="z-punctuation z-section z-mapping-or-set z-begin z-python">{</span> </span></span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-meta z-mapping-or-set z-python"></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">Name<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">ProcessChurnData<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">ActionOnFailure<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">CONTINUE<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">HadoopJarStep<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-mapping-or-set z-python"><span class="z-punctuation z-section z-mapping-or-set z-begin z-python">{</span> </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-mapping-or-set z-python"></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">Jar<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">command-runner.jar<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">Args<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-punctuation z-section z-sequence z-begin z-python">[</span> </span></span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">spark-submit<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> </span></span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">--deploy-mode<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">cluster<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> </span></span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">--master<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">yarn<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> </span></span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">--conf<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">spark.sql.adaptive.enabled=true<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> </span></span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">s3://company-spark-jobs/churn_data_processing.py<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> </span></span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">--input-date<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-constant z-character z-escape z-python">{{</span> ds <span class="z-constant z-character z-escape z-python">}}</span><span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> </span></span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">--output-path<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">s3://company-processed-data/churn_features/<span class="z-constant z-character z-escape z-python">{{</span> ds <span class="z-constant z-character z-escape z-python">}}</span>/<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span> </span></span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"> <span class="z-punctuation z-section z-sequence z-end z-python">]</span></span> </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-mapping z-value z-python"></span><span class="z-meta z-mapping z-python"> <span class="z-punctuation z-section z-mapping z-end z-python">}</span></span> </span></span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-meta z-mapping z-value z-python"></span><span class="z-meta z-mapping z-python"> <span class="z-punctuation z-section z-mapping z-end z-python">}</span></span> </span></span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-sequence z-list z-python"> <span class="z-punctuation z-section z-sequence z-end z-python">]</span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-mapping z-python"><span class="z-punctuation z-section z-mapping z-end z-python">}</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">create_emr_cluster</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">EmrCreateJobFlowOperator</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">task_id</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">create_emr_cluster<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">job_flow_overrides</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-other z-constant z-python">EMR_CONFIG</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">aws_conn_id</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">aws_default<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">dag</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">dag</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">wait_for_spark_job</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">EmrJobFlowSensor</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">task_id</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">wait_for_spark_job<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">job_flow_id</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-constant z-character z-escape z-python">{{</span> task_instance.xcom_pull(task_ids=&#39;create_emr_cluster&#39;, key=&#39;return_value&#39;) <span class="z-constant z-character z-escape z-python">}}</span><span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">aws_conn_id</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">aws_default<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">dag</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">dag</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">terminate_emr_cluster</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">EmrTerminateJobFlowOperator</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">task_id</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">terminate_emr_cluster<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">job_flow_id</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-constant z-character z-escape z-python">{{</span> task_instance.xcom_pull(task_ids=&#39;create_emr_cluster&#39;, key=&#39;return_value&#39;) <span class="z-constant z-character z-escape z-python">}}</span><span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">aws_conn_id</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">aws_default<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">trigger_rule</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">TriggerRule</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-variable z-other z-constant z-python">ALL_DONE</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">dag</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">dag</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Task 4: Train ML model </span></span><span class="z-source z-python"><span class="z-meta z-function z-python"><span class="z-storage z-type z-function z-python"><span class="z-keyword z-declaration z-function z-python">def</span></span> <span class="z-entity z-name z-function z-python"><span class="z-meta z-generic-name z-python">train_churn_model</span></span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-begin z-python">(</span></span><span class="z-meta z-function z-parameters z-python"><span class="z-keyword z-operator z-unpacking z-mapping z-python">**</span><span class="z-variable z-parameter z-python">context</span><span class="z-punctuation z-section z-parameters z-end z-python">)</span></span><span class="z-meta z-function z-python"><span class="z-punctuation z-section z-function z-begin z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-comment z-block z-documentation z-python"><span class="z-punctuation z-definition z-comment z-begin z-python">&quot;&quot;&quot;</span>Train customer churn prediction model<span class="z-punctuation z-definition z-comment z-end z-python">&quot;&quot;&quot;</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">execution_date</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-item-access z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">context</span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-begin z-python">[</span></span><span class="z-meta z-item-access z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">execution_date<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-end z-python">]</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">date_str</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">execution_date</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">strftime</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-constant z-other z-placeholder z-python">%Y</span>-<span class="z-constant z-other z-placeholder z-python">%m</span>-<span class="z-constant z-other z-placeholder z-python">%d</span><span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Load processed features from S3 </span></span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">s3</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">boto3</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">client</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">s3<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">feature_path</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-storage z-type z-string z-python">f</span><span class="z-meta z-string z-interpolated z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-interpolated z-python"><span class="z-string z-quoted z-single z-python">s3://company-processed-data/churn_features/</span><span class="z-meta z-interpolation z-python"><span class="z-punctuation z-section z-interpolation z-begin z-python">{</span><span class="z-source z-python z-embedded"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">date_str</span></span></span></span><span class="z-meta z-interpolation z-python"><span class="z-punctuation z-section z-interpolation z-end z-python">}</span></span><span class="z-string z-quoted z-single z-python">/features.parquet<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> In production, you&#39;d read actual data </span></span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> df = pd.read_parquet(feature_path) </span></span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Simulated training for demonstration </span></span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">print</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">Loading training data...<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> X_train, y_train = df.drop(&#39;churned&#39;, axis=1), df[&#39;churned&#39;] </span></span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">print</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">Training Random Forest model...<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">model</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">RandomForestClassifier</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">n_estimators</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-constant z-numeric z-integer z-decimal z-python">100</span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">max_depth</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-constant z-numeric z-integer z-decimal z-python">10</span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">min_samples_split</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-constant z-numeric z-integer z-decimal z-python">5</span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">random_state</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-constant z-numeric z-integer z-decimal z-python">42</span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">n_jobs</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-keyword z-operator z-arithmetic z-python">-</span><span class="z-constant z-numeric z-integer z-decimal z-python">1</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> model.fit(X_train, y_train) </span></span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Save model to S3 </span></span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">model_path</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-storage z-type z-string z-python">f</span><span class="z-meta z-string z-interpolated z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-interpolated z-python"><span class="z-string z-quoted z-single z-python">/tmp/churn_model_</span><span class="z-meta z-interpolation z-python"><span class="z-punctuation z-section z-interpolation z-begin z-python">{</span><span class="z-source z-python z-embedded"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">date_str</span></span></span></span><span class="z-meta z-interpolation z-python"><span class="z-punctuation z-section z-interpolation z-end z-python">}</span></span><span class="z-string z-quoted z-single z-python">.pkl<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">joblib</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">dump</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">model</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">model_path</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">s3</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">upload_file</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">model_path</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">company-ml-models<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-storage z-type z-string z-python">f</span><span class="z-meta z-string z-interpolated z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-interpolated z-python"><span class="z-string z-quoted z-single z-python">churn_prediction/</span><span class="z-meta z-interpolation z-python"><span class="z-punctuation z-section z-interpolation z-begin z-python">{</span><span class="z-source z-python z-embedded"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">date_str</span></span></span></span><span class="z-meta z-interpolation z-python"><span class="z-punctuation z-section z-interpolation z-end z-python">}</span></span><span class="z-string z-quoted z-single z-python">/model.pkl<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">print</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-storage z-type z-string z-python">f</span><span class="z-meta z-string z-interpolated z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-interpolated z-python"><span class="z-string z-quoted z-double z-python">Model trained and saved to S3<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Store model path in XCom for next task </span></span><span class="z-source z-python"> <span class="z-meta z-item-access z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">context</span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-begin z-python">[</span></span><span class="z-meta z-item-access z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">task_instance<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-end z-python">]</span></span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">xcom_push</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-variable z-parameter z-python">key</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">model_path<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">value</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-storage z-type z-string z-python">f</span><span class="z-meta z-string z-interpolated z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-interpolated z-python"><span class="z-string z-quoted z-single z-python">churn_prediction/</span><span class="z-meta z-interpolation z-python"><span class="z-punctuation z-section z-interpolation z-begin z-python">{</span><span class="z-source z-python z-embedded"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">date_str</span></span></span></span><span class="z-meta z-interpolation z-python"><span class="z-punctuation z-section z-interpolation z-end z-python">}</span></span><span class="z-string z-quoted z-single z-python">/model.pkl<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-keyword z-control z-flow z-return z-python">return</span> <span class="z-constant z-language z-python">True</span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">train_model_task</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">PythonOperator</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">task_id</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">train_model<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">python_callable</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">train_churn_model</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">provide_context</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-constant z-language z-python">True</span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">dag</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">dag</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Task 5: Validate model performance </span></span><span class="z-source z-python"><span class="z-meta z-function z-python"><span class="z-storage z-type z-function z-python"><span class="z-keyword z-declaration z-function z-python">def</span></span> <span class="z-entity z-name z-function z-python"><span class="z-meta z-generic-name z-python">validate_model</span></span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-begin z-python">(</span></span><span class="z-meta z-function z-parameters z-python"><span class="z-keyword z-operator z-unpacking z-mapping z-python">**</span><span class="z-variable z-parameter z-python">context</span><span class="z-punctuation z-section z-parameters z-end z-python">)</span></span><span class="z-meta z-function z-python"><span class="z-punctuation z-section z-function z-begin z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-comment z-block z-documentation z-python"><span class="z-punctuation z-definition z-comment z-begin z-python">&quot;&quot;&quot;</span>Validate model performance on test set<span class="z-punctuation z-definition z-comment z-end z-python">&quot;&quot;&quot;</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">execution_date</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-item-access z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">context</span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-begin z-python">[</span></span><span class="z-meta z-item-access z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">execution_date<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-end z-python">]</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">date_str</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">execution_date</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">strftime</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-constant z-other z-placeholder z-python">%Y</span>-<span class="z-constant z-other z-placeholder z-python">%m</span>-<span class="z-constant z-other z-placeholder z-python">%d</span><span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Load model from S3 </span></span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">s3</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">boto3</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">client</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">s3<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">model_path</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-item-access z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">context</span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-begin z-python">[</span></span><span class="z-meta z-item-access z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">task_instance<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-end z-python">]</span></span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">xcom_pull</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">task_ids</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">train_model<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">key</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">model_path<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Download and load model </span></span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">s3</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">download_file</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">company-ml-models<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">model_path</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">/tmp/model.pkl<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">model</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">joblib</span></span><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">load</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">/tmp/model.pkl<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Load test data </span></span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> In production, load actual test data </span></span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> X_test, y_test = load_test_data() </span></span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Make predictions (simulated) </span></span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> y_pred = model.predict(X_test) </span></span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Calculate metrics (simulated values) </span></span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">metrics</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-mapping-or-set z-python"><span class="z-punctuation z-section z-mapping-or-set z-begin z-python">{</span> </span></span><span class="z-source z-python"><span class="z-meta z-mapping-or-set z-python"></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">accuracy<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-constant z-numeric z-float z-decimal z-python">0<span class="z-punctuation z-separator z-decimal z-python">.</span>87</span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">precision<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-constant z-numeric z-float z-decimal z-python">0<span class="z-punctuation z-separator z-decimal z-python">.</span>82</span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">recall<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-constant z-numeric z-float z-decimal z-python">0<span class="z-punctuation z-separator z-decimal z-python">.</span>79</span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">f1_score<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-constant z-numeric z-float z-decimal z-python">0<span class="z-punctuation z-separator z-decimal z-python">.</span>80</span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-key z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">date<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-mapping z-python"><span class="z-punctuation z-separator z-mapping z-key-value z-python">:</span></span><span class="z-meta z-mapping z-python"> </span><span class="z-meta z-mapping z-value z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">date_str</span></span> </span></span><span class="z-source z-python"><span class="z-meta z-mapping z-value z-python"></span><span class="z-meta z-mapping z-python"> <span class="z-punctuation z-section z-mapping z-end z-python">}</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">print</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-storage z-type z-string z-python">f</span><span class="z-meta z-string z-interpolated z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-interpolated z-python"><span class="z-string z-quoted z-double z-python">Model Metrics: </span><span class="z-meta z-interpolation z-python"><span class="z-punctuation z-section z-interpolation z-begin z-python">{</span><span class="z-source z-python z-embedded"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">metrics</span></span></span></span><span class="z-meta z-interpolation z-python"><span class="z-punctuation z-section z-interpolation z-end z-python">}</span></span><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Store metrics in database </span></span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> In production, write to actual database </span></span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Store metrics in XCom </span></span><span class="z-source z-python"> <span class="z-meta z-item-access z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">context</span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-begin z-python">[</span></span><span class="z-meta z-item-access z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">task_instance<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-end z-python">]</span></span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">xcom_push</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-variable z-parameter z-python">key</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">metrics<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> <span class="z-variable z-parameter z-python">value</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">metrics</span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Return branch decision based on performance </span></span><span class="z-source z-python"> <span class="z-meta z-statement z-conditional z-if z-python"><span class="z-keyword z-control z-conditional z-if z-python">if</span> <span class="z-meta z-item-access z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">metrics</span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-begin z-python">[</span></span><span class="z-meta z-item-access z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">f1_score<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-end z-python">]</span></span> <span class="z-keyword z-operator z-comparison z-python">&gt;=</span> <span class="z-constant z-numeric z-float z-decimal z-python">0<span class="z-punctuation z-separator z-decimal z-python">.</span>75</span><span class="z-punctuation z-section z-block z-conditional z-if z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-keyword z-control z-flow z-return z-python">return</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">deploy_model<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span> </span><span class="z-source z-python"> <span class="z-meta z-statement z-conditional z-else z-python"><span class="z-keyword z-control z-conditional z-else z-python">else</span><span class="z-punctuation z-section z-block z-conditional z-else z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-keyword z-control z-flow z-return z-python">return</span> <span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">alert_poor_performance<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">validate_model_task</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">BranchPythonOperator</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">task_id</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">validate_model<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">python_callable</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">validate_model</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">provide_context</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-constant z-language z-python">True</span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">dag</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">dag</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Task 6a: Deploy model if performance is good </span></span><span class="z-source z-python"><span class="z-meta z-function z-python"><span class="z-storage z-type z-function z-python"><span class="z-keyword z-declaration z-function z-python">def</span></span> <span class="z-entity z-name z-function z-python"><span class="z-meta z-generic-name z-python">deploy_model</span></span></span><span class="z-meta z-function z-parameters z-python"><span class="z-punctuation z-section z-parameters z-begin z-python">(</span></span><span class="z-meta z-function z-parameters z-python"><span class="z-keyword z-operator z-unpacking z-mapping z-python">**</span><span class="z-variable z-parameter z-python">context</span><span class="z-punctuation z-section z-parameters z-end z-python">)</span></span><span class="z-meta z-function z-python"><span class="z-punctuation z-section z-function z-begin z-python">:</span></span> </span><span class="z-source z-python"> <span class="z-comment z-block z-documentation z-python"><span class="z-punctuation z-definition z-comment z-begin z-python">&quot;&quot;&quot;</span>Deploy model to production serving endpoint<span class="z-punctuation z-definition z-comment z-end z-python">&quot;&quot;&quot;</span></span> </span><span class="z-source z-python"> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">model_path</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-item-access z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">context</span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-begin z-python">[</span></span><span class="z-meta z-item-access z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">task_instance<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span></span><span class="z-meta z-item-access z-python"><span class="z-punctuation z-section z-brackets z-end z-python">]</span></span><span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-punctuation z-accessor z-dot z-python">.</span></span><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">xcom_pull</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">task_ids</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">train_model<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">key</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">model_path<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Deploy to SageMaker/Kubernetes/Model serving platform </span></span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">print</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-storage z-type z-string z-python">f</span><span class="z-meta z-string z-interpolated z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-interpolated z-python"><span class="z-string z-quoted z-double z-python">Deploying model from </span><span class="z-meta z-interpolation z-python"><span class="z-punctuation z-section z-interpolation z-begin z-python">{</span><span class="z-source z-python z-embedded"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">model_path</span></span></span></span><span class="z-meta z-interpolation z-python"><span class="z-punctuation z-section z-interpolation z-end z-python">}</span></span><span class="z-string z-quoted z-double z-python"> to production...<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Update model registry </span></span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Update feature store </span></span><span class="z-source z-python"> <span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Update model serving endpoint </span></span><span class="z-source z-python"> </span><span class="z-source z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">print</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">Model successfully deployed to production<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> <span class="z-keyword z-control z-flow z-return z-python">return</span> <span class="z-constant z-language z-python">True</span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">deploy_model_task</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">PythonOperator</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">task_id</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">deploy_model<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">python_callable</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">deploy_model</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">provide_context</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-constant z-language z-python">True</span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">dag</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">dag</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Task 6b: Alert if performance is poor </span></span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">alert_poor_performance</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">SlackWebhookOperator</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">task_id</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">alert_poor_performance<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">http_conn_id</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">slack_webhook<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">message</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;&quot;&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"> </span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"> <span class="z-meta z-string z-python"></span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python">:warning: *Model Performance Alert* </span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"> </span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"> The churn prediction model for <span class="z-constant z-character z-escape z-python">{{</span> ds <span class="z-constant z-character z-escape z-python">}}</span> did not meet performance thresholds. </span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"> </span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"> *Metrics:* </span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"> <span class="z-constant z-character z-escape z-python">{{</span> task_instance.xcom_pull(task_ids=&#39;validate_model&#39;, key=&#39;metrics&#39;) <span class="z-constant z-character z-escape z-python">}}</span> </span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"> </span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"> Please review the model and data quality. </span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"> <span class="z-punctuation z-definition z-string z-end z-python">&quot;&quot;&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">dag</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">dag</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Task 7: Update model registry </span></span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">update_registry</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">PostgresOperator</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">task_id</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">update_model_registry<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">postgres_conn_id</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">postgres_ml_metadata<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">sql</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;&quot;&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"> </span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-meta z-string z-python"></span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-source z-sql"> <span class="z-keyword z-other z-DML z-sql">INSERT INTO</span> model_registry ( </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-source z-sql"> model_name, </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-source z-sql"> version, </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-source z-sql"> metrics, </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-source z-sql"> model_path, </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-source z-sql"> created_at, </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-source z-sql"> status </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-source z-sql"> ) <span class="z-keyword z-other z-DML z-II z-sql">VALUES</span> ( </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-source z-sql"> <span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>churn_prediction<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span>, </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-source z-sql"> <span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span><span class="z-constant z-character z-escape z-python">{{</span> ds <span class="z-constant z-character z-escape z-python">}}</span><span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span>, </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-source z-sql"> <span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span><span class="z-constant z-character z-escape z-python">{{</span> task_instance.xcom_pull(task_ids=&quot;validate_model&quot;, key=&quot;metrics&quot;) | tojson <span class="z-constant z-character z-escape z-python">}}</span><span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span>, </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-source z-sql"> <span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span><span class="z-constant z-character z-escape z-python">{{</span> task_instance.xcom_pull(task_ids=&quot;train_model&quot;, key=&quot;model_path&quot;) <span class="z-constant z-character z-escape z-python">}}</span><span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span>, </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-source z-sql"> <span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span><span class="z-constant z-character z-escape z-python">{{</span> ts <span class="z-constant z-character z-escape z-python">}}</span><span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span>, </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-source z-sql"> <span class="z-string z-quoted z-single z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&#39;</span>deployed<span class="z-punctuation z-definition z-string z-end z-sql">&#39;</span></span> </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-source z-sql"> ); </span></span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-source z-sql"> </span><span class="z-punctuation z-definition z-string z-end z-python">&quot;&quot;&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">trigger_rule</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">TriggerRule</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-variable z-other z-constant z-python">ONE_SUCCESS</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">dag</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">dag</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Task 8: Send success notification </span></span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">send_success_notification</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">SlackWebhookOperator</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">task_id</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">send_success_notification<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">http_conn_id</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">slack_webhook<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">message</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;&quot;&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"> </span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"> <span class="z-meta z-string z-python"></span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python">:white_check_mark: *ML Pipeline Success* </span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"> </span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"> The churn prediction pipeline for <span class="z-constant z-character z-escape z-python">{{</span> ds <span class="z-constant z-character z-escape z-python">}}</span> completed successfully! </span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"> </span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"> *Metrics:* </span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"> <span class="z-constant z-character z-escape z-python">{{</span> task_instance.xcom_pull(task_ids=&#39;validate_model&#39;, key=&#39;metrics&#39;) <span class="z-constant z-character z-escape z-python">}}</span> </span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"> </span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"> Model deployed and ready for inference. </span></span></span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-block z-python"> <span class="z-punctuation z-definition z-string z-end z-python">&quot;&quot;&quot;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">trigger_rule</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">TriggerRule</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-variable z-other z-constant z-python">ONE_SUCCESS</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">dag</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">dag</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Task 9: Cleanup temporary files </span></span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">cleanup</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">BashOperator</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">task_id</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">cleanup<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">bash_command</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">rm -rf /tmp/churn_* /tmp/model.pkl<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">trigger_rule</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">TriggerRule</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-variable z-other z-constant z-python">ALL_DONE</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">dag</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">dag</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Task 10: Generate daily report </span></span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">generate_report</span></span> <span class="z-keyword z-operator z-assignment z-python">=</span> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-meta z-generic-name z-python">PythonOperator</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">task_id</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&#39;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-single z-python">generate_daily_report<span class="z-punctuation z-definition z-string z-end z-python">&#39;</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">python_callable</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-function z-inline z-python"><span class="z-storage z-type z-function z-inline z-python"><span class="z-keyword z-declaration z-function z-inline z-python">lambda</span></span></span><span class="z-meta z-function z-inline z-python"><span class="z-meta z-function z-inline z-parameters z-python"> <span class="z-keyword z-operator z-unpacking z-mapping z-python">**</span><span class="z-variable z-parameter z-python">context</span><span class="z-punctuation z-section z-function z-begin z-python">:</span></span></span><span class="z-meta z-function z-inline z-body z-python"> <span class="z-meta z-function-call z-python"><span class="z-meta z-qualified-name z-python"><span class="z-variable z-function z-python"><span class="z-support z-function z-builtin z-python">print</span></span></span></span><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-begin z-python">(</span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python"><span class="z-punctuation z-definition z-string z-begin z-python">&quot;</span></span></span><span class="z-meta z-string z-python"><span class="z-string z-quoted z-double z-python">Generating daily report...<span class="z-punctuation z-definition z-string z-end z-python">&quot;</span></span></span><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">provide_context</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-constant z-language z-python">True</span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">trigger_rule</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">TriggerRule</span><span class="z-punctuation z-accessor z-dot z-python">.</span><span class="z-variable z-other z-constant z-python">ALL_DONE</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"> <span class="z-variable z-parameter z-python">dag</span><span class="z-keyword z-operator z-assignment z-python">=</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">dag</span></span><span class="z-punctuation z-separator z-arguments z-python">,</span> </span></span><span class="z-source z-python"><span class="z-meta z-function-call z-arguments z-python"><span class="z-punctuation z-section z-arguments z-end z-python">)</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Define task dependencies </span></span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">check_data_task</span></span> <span class="z-keyword z-operator z-comparison z-python">&gt;</span><span class="z-keyword z-operator z-comparison z-python">&gt;</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">data_quality_task</span></span> <span class="z-keyword z-operator z-comparison z-python">&gt;</span><span class="z-keyword z-operator z-comparison z-python">&gt;</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">create_emr_cluster</span></span> </span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">create_emr_cluster</span></span> <span class="z-keyword z-operator z-comparison z-python">&gt;</span><span class="z-keyword z-operator z-comparison z-python">&gt;</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">wait_for_spark_job</span></span> <span class="z-keyword z-operator z-comparison z-python">&gt;</span><span class="z-keyword z-operator z-comparison z-python">&gt;</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">terminate_emr_cluster</span></span> </span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">terminate_emr_cluster</span></span> <span class="z-keyword z-operator z-comparison z-python">&gt;</span><span class="z-keyword z-operator z-comparison z-python">&gt;</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">train_model_task</span></span> <span class="z-keyword z-operator z-comparison z-python">&gt;</span><span class="z-keyword z-operator z-comparison z-python">&gt;</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">validate_model_task</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Branching based on model performance </span></span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">validate_model_task</span></span> <span class="z-keyword z-operator z-comparison z-python">&gt;</span><span class="z-keyword z-operator z-comparison z-python">&gt;</span> <span class="z-meta z-sequence z-list z-python"><span class="z-punctuation z-section z-sequence z-begin z-python">[</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">deploy_model_task</span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">alert_poor_performance</span></span><span class="z-punctuation z-section z-sequence z-end z-python">]</span></span> </span><span class="z-source z-python"> </span><span class="z-source z-python"><span class="z-comment z-line z-number-sign z-python"><span class="z-punctuation z-definition z-comment z-python">#</span> Both branches converge to update registry </span></span><span class="z-source z-python"><span class="z-meta z-sequence z-list z-python"><span class="z-punctuation z-section z-sequence z-begin z-python">[</span><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">deploy_model_task</span></span><span class="z-punctuation z-separator z-sequence z-python">,</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">alert_poor_performance</span></span><span class="z-punctuation z-section z-sequence z-end z-python">]</span></span> <span class="z-keyword z-operator z-comparison z-python">&gt;</span><span class="z-keyword z-operator z-comparison z-python">&gt;</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">update_registry</span></span> </span><span class="z-source z-python"><span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">update_registry</span></span> <span class="z-keyword z-operator z-comparison z-python">&gt;</span><span class="z-keyword z-operator z-comparison z-python">&gt;</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">send_success_notification</span></span> <span class="z-keyword z-operator z-comparison z-python">&gt;</span><span class="z-keyword z-operator z-comparison z-python">&gt;</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">cleanup</span></span> <span class="z-keyword z-operator z-comparison z-python">&gt;</span><span class="z-keyword z-operator z-comparison z-python">&gt;</span> <span class="z-meta z-qualified-name z-python"><span class="z-meta z-generic-name z-python">generate_report</span></span> </span></code></pre> <p><strong>Why Airflow for this use case?</strong></p> <ul> <li><strong>Orchestrates complex workflows</strong>: Manages dependencies between multiple tasks</li> <li><strong>Handles failures gracefully</strong>: Automatic retries and alerting</li> <li><strong>Dynamic pipelines</strong>: Can generate tasks based on runtime conditions</li> <li><strong>Monitors progress</strong>: Rich UI to track pipeline execution</li> <li><strong>Integrations</strong>: Connects with Spark, databases, cloud services, etc.</li> <li><strong>Scheduling</strong>: Runs pipelines on schedule with dependency management</li> </ul> <h2 id="comparison-table">Comparison Table<a class="zola-anchor" href="#comparison-table" aria-label="Anchor link for: comparison-table" style="visibility: hidden;"></a> </h2> <table><thead><tr><th>Feature</th><th>Apache Spark</th><th>Apache Flink</th><th>Apache Airflow</th></tr></thead><tbody> <tr><td><strong>Primary Purpose</strong></td><td>Batch &amp; Streaming Processing</td><td>Stream Processing</td><td>Workflow Orchestration</td></tr> <tr><td><strong>Processing Model</strong></td><td>Micro-batch</td><td>True streaming</td><td>N/A (orchestrator)</td></tr> <tr><td><strong>Latency</strong></td><td>Seconds</td><td>Milliseconds</td><td>N/A</td></tr> <tr><td><strong>State Management</strong></td><td>Limited</td><td>Excellent</td><td>N/A</td></tr> <tr><td><strong>Fault Tolerance</strong></td><td>RDD lineage, checkpointing</td><td>Checkpointing, savepoints</td><td>Task retries</td></tr> <tr><td><strong>APIs</strong></td><td>Java, Scala, Python, R, SQL</td><td>Java, Scala, Python, SQL</td><td>Python</td></tr> <tr><td><strong>Learning Curve</strong></td><td>Moderate</td><td>Steep</td><td>Easy to Moderate</td></tr> <tr><td><strong>Maturity</strong></td><td>Very mature</td><td>Mature</td><td>Very mature</td></tr> <tr><td><strong>Community</strong></td><td>Very large</td><td>Large</td><td>Very large</td></tr> <tr><td><strong>Use Cases</strong></td><td>Batch analytics, ML, ETL</td><td>Real-time analytics, CEP</td><td>Pipeline orchestration</td></tr> <tr><td><strong>Memory Usage</strong></td><td>High (in-memory)</td><td>Moderate</td><td>Low</td></tr> <tr><td><strong>Exactly-Once</strong></td><td>Structured Streaming</td><td>Native support</td><td>Via operators</td></tr> <tr><td><strong>Event Time</strong></td><td>Watermarks</td><td>Native support</td><td>N/A</td></tr> <tr><td><strong>SQL Support</strong></td><td>Excellent (Spark SQL)</td><td>Good (Flink SQL)</td><td>Via operators</td></tr> <tr><td><strong>ML Libraries</strong></td><td>MLlib (built-in)</td><td>FlinkML (limited)</td><td>Integrates external</td></tr> <tr><td><strong>Deployment</strong></td><td>Standalone, YARN, K8s, Mesos</td><td>Standalone, YARN, K8s, Mesos</td><td>Standalone, K8s, Docker</td></tr> <tr><td><strong>Monitoring</strong></td><td>Spark UI, metrics</td><td>Flink UI, metrics</td><td>Rich web UI</td></tr> </tbody></table> <h2 id="when-to-use-which-tool">When to Use Which Tool?<a class="zola-anchor" href="#when-to-use-which-tool" aria-label="Anchor link for: when-to-use-which-tool" style="visibility: hidden;"></a> </h2> <h3 id="use-apache-spark-when">Use Apache Spark When:<a class="zola-anchor" href="#use-apache-spark-when" aria-label="Anchor link for: use-apache-spark-when" style="visibility: hidden;"></a> </h3> <ol> <li><strong>Batch processing of large datasets</strong>: Historical data analysis, ETL jobs</li> <li><strong>Complex analytics</strong>: Machine learning, graph processing, SQL queries</li> <li><strong>Near real-time is acceptable</strong>: Micro-batch processing with seconds latency</li> <li><strong>You need mature ML libraries</strong>: MLlib provides comprehensive algorithms</li> <li><strong>Flexibility in languages</strong>: Need support for multiple programming languages</li> <li><strong>Cost optimization</strong>: Can use spot instances for batch jobs</li> </ol> <p><strong>Examples:</strong></p> <ul> <li>Daily report generation</li> <li>Training machine learning models</li> <li>Data warehouse ETL</li> <li>Log analysis and aggregation</li> <li>Feature engineering for ML</li> </ul> <h3 id="use-apache-flink-when">Use Apache Flink When:<a class="zola-anchor" href="#use-apache-flink-when" aria-label="Anchor link for: use-apache-flink-when" style="visibility: hidden;"></a> </h3> <ol> <li><strong>True real-time processing</strong>: Millisecond-level latency required</li> <li><strong>Complex event processing</strong>: Detecting patterns across multiple events</li> <li><strong>Stateful computations</strong>: Need to maintain state across events</li> <li><strong>Event time processing</strong>: Handling out-of-order events correctly</li> <li><strong>Exactly-once guarantees</strong>: Critical for financial or compliance applications</li> <li><strong>Continuous queries</strong>: Long-running streaming queries</li> </ol> <p><strong>Examples:</strong></p> <ul> <li>Fraud detection</li> <li>Real-time recommendations</li> <li>Network monitoring and anomaly detection</li> <li>Real-time dashboards</li> <li>IoT event processing</li> <li>Stock trading systems</li> </ul> <h3 id="use-apache-airflow-when">Use Apache Airflow When:<a class="zola-anchor" href="#use-apache-airflow-when" aria-label="Anchor link for: use-apache-airflow-when" style="visibility: hidden;"></a> </h3> <ol> <li><strong>Orchestrating complex workflows</strong>: Multiple dependent tasks</li> <li><strong>Scheduling data pipelines</strong>: Run pipelines on a schedule</li> <li><strong>Managing dependencies</strong>: Tasks that depend on other tasks or external systems</li> <li><strong>Monitoring and alerting</strong>: Need visibility into pipeline execution</li> <li><strong>Mixing multiple tools</strong>: Coordinating Spark, Flink, databases, APIs, etc.</li> <li><strong>Dynamic pipelines</strong>: Generate tasks based on runtime conditions</li> </ol> <p><strong>Examples:</strong></p> <ul> <li>ETL pipelines</li> <li>ML model training pipelines</li> <li>Data quality monitoring</li> <li>Report generation workflows</li> <li>Multi-stage data processing</li> <li>Backup and maintenance tasks</li> </ul> <h2 id="can-you-use-them-together">Can You Use Them Together?<a class="zola-anchor" href="#can-you-use-them-together" aria-label="Anchor link for: can-you-use-them-together" style="visibility: hidden;"></a> </h2> <p><strong>Absolutely!</strong> In fact, many production data platforms use all three together:</p> <p><strong>Example Architecture:</strong></p> <ol> <li><strong>Airflow</strong> orchestrates the entire data platform</li> <li><strong>Flink</strong> processes real-time streaming data</li> <li><strong>Spark</strong> runs batch jobs for historical analysis and ML training</li> <li><strong>Airflow</strong> schedules when Spark jobs run</li> <li><strong>Airflow</strong> monitors Flink streaming jobs</li> <li><strong>Airflow</strong> triggers alerts if either fails</li> </ol> <p><strong>Real-world scenario:</strong></p> <pre class="z-code"><code><span class="z-text z-plain">Daily at 2 AM (Airflow scheduled): </span><span class="z-text z-plain"> → Extract data from APIs (Airflow task) </span><span class="z-text z-plain"> → Run data quality checks (Airflow task) </span><span class="z-text z-plain"> → Process with Spark (Airflow triggers Spark job) </span><span class="z-text z-plain"> → Train ML model (Airflow triggers Spark ML job) </span><span class="z-text z-plain"> → Deploy model (Airflow task) </span><span class="z-text z-plain"> </span><span class="z-text z-plain">Continuous (Flink streaming): </span><span class="z-text z-plain"> → Read events from Kafka </span><span class="z-text z-plain"> → Apply ML model for real-time predictions </span><span class="z-text z-plain"> → Write results to database </span><span class="z-text z-plain"> → Send alerts on anomalies </span><span class="z-text z-plain"> </span><span class="z-text z-plain">Hourly (Airflow scheduled): </span><span class="z-text z-plain"> → Check Flink job health (Airflow sensor) </span><span class="z-text z-plain"> → Restart Flink job if failed (Airflow task) </span><span class="z-text z-plain"> → Update monitoring dashboard (Airflow task) </span></code></pre> <h2 id="conclusion">Conclusion<a class="zola-anchor" href="#conclusion" aria-label="Anchor link for: conclusion" style="visibility: hidden;"></a> </h2> <p>Choosing between Apache Spark, Flink, and Airflow depends on your specific use case:</p> <ul> <li><strong>Spark</strong> is your go-to for batch processing and complex analytics with a rich ecosystem</li> <li><strong>Flink</strong> excels at true real-time stream processing with low latency and stateful computations</li> <li><strong>Airflow</strong> is essential for orchestrating and managing complex data workflows</li> </ul> <p>In modern data platforms, you'll often use all three:</p> <ul> <li><strong>Airflow</strong> as the orchestrator managing the entire data platform</li> <li><strong>Flink</strong> handling real-time event streams</li> <li><strong>Spark</strong> processing large-scale batch jobs and machine learning workloads</li> </ul> <p>The key is understanding that these tools complement each other rather than compete. By using them together strategically, you can build robust, scalable data platforms that handle both real-time and batch processing needs while maintaining clear visibility and control over your workflows.</p> <h2 id="further-reading">Further Reading<a class="zola-anchor" href="#further-reading" aria-label="Anchor link for: further-reading" style="visibility: hidden;"></a> </h2> <ul> <li><a rel="noopener nofollow noreferrer" target="_blank" href="https://spark.apache.org/docs/latest/">Apache Spark Documentation</a></li> <li><a rel="noopener nofollow noreferrer" target="_blank" href="https://flink.apache.org/docs/stable/">Apache Flink Documentation</a></li> <li><a rel="noopener nofollow noreferrer" target="_blank" href="https://airflow.apache.org/docs/">Apache Airflow Documentation</a></li> <li><a rel="noopener nofollow noreferrer" target="_blank" href="https://www.oreilly.com/library/view/stream-processing-with/9781491974285/">Stream Processing with Apache Flink (O'Reilly)</a></li> <li><a rel="noopener nofollow noreferrer" target="_blank" href="https://www.oreilly.com/library/view/learning-spark-2nd/9781492050032/">Learning Spark, 2nd Edition (O'Reilly)</a></li> <li><a rel="noopener nofollow noreferrer" target="_blank" href="https://www.manning.com/books/data-pipelines-with-apache-airflow">Data Pipelines with Apache Airflow (Manning)</a></li> </ul> 2025-12-11T00:00:00+00:00 2025-12-11T00:00:00+00:00 https://rajnandan.com/posts/memoization-optimization-typescript/ <h2 id="the-core-insight">The Core Insight<a class="zola-anchor" href="#the-core-insight" aria-label="Anchor link for: the-core-insight" style="visibility: hidden;"></a> </h2> <p>Memoization is elegantly simple: cache a function's results by its inputs. When the same input appears again, return the cached value instead of recomputing. You're trading memory for speed - replacing repeated work with instant lookups.</p> <h2 id="the-problem-counting-paths-up-stairs">The Problem: Counting Paths Up Stairs<a class="zola-anchor" href="#the-problem-counting-paths-up-stairs" aria-label="Anchor link for: the-problem-counting-paths-up-stairs" style="visibility: hidden;"></a> </h2> <p>Imagine a staircase where you can climb 1, 3, or 5 steps at a time. How many distinct ways can you reach step <code>n</code>?</p> <p>Let <code>f(n)</code> be the count of ways. The recurrence relation is:</p> <pre class="z-code"><code><span class="z-text z-plain">f(n) = f(n-1) + f(n-3) + f(n-5) </span></code></pre> <p>With base cases:</p> <ul> <li><code>f(0) = 1</code> — one valid way: do nothing</li> <li><code>f(n) = 0</code> when <code>n &lt; 0</code> — invalid path</li> </ul> <h2 id="naive-recursion-clean-but-catastrophically-slow">Naive Recursion: Clean but Catastrophically Slow<a class="zola-anchor" href="#naive-recursion-clean-but-catastrophically-slow" aria-label="Anchor link for: naive-recursion-clean-but-catastrophically-slow" style="visibility: hidden;"></a> </h2> <pre data-lang="typescript" class="language-typescript z-code"><code class="language-typescript" data-lang="typescript"><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-storage z-type z-function z-ts">function</span> <span class="z-meta z-definition z-function z-ts"><span class="z-entity z-name z-function z-ts">stepCount</span></span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">n</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-support z-type z-primitive z-ts">number</span></span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span><span class="z-meta z-return z-type z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-support z-type z-primitive z-ts">number</span> </span><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">n</span> <span class="z-keyword z-operator z-relational z-ts">&lt;</span> <span class="z-constant z-numeric z-decimal z-ts">0</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-constant z-numeric z-decimal z-ts">0</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">n</span> <span class="z-keyword z-operator z-comparison z-ts">===</span> <span class="z-constant z-numeric z-decimal z-ts">0</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-constant z-numeric z-decimal z-ts">1</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">stepCount</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">n</span> <span class="z-keyword z-operator z-arithmetic z-ts">-</span> <span class="z-constant z-numeric z-decimal z-ts">1</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-operator z-arithmetic z-ts">+</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">stepCount</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">n</span> <span class="z-keyword z-operator z-arithmetic z-ts">-</span> <span class="z-constant z-numeric z-decimal z-ts">3</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-operator z-arithmetic z-ts">+</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">stepCount</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">n</span> <span class="z-keyword z-operator z-arithmetic z-ts">-</span> <span class="z-constant z-numeric z-decimal z-ts">5</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></code></pre> <p>This reads like the mathematical definition. Beautiful, right?</p> <p><strong>The problem</strong>: identical subproblems explode across branches. For <code>n = 10</code>, <code>stepCount(5)</code> gets called multiple times. The call tree grows exponentially, and we keep solving the same problems over and over.</p> <p>Try <code>stepCount(35)</code> and watch your CPU fan spin up. For <code>n = 50</code>, you might as well go make coffee.</p> <h2 id="memoization-add-a-memory">Memoization: Add a Memory<a class="zola-anchor" href="#memoization-add-a-memory" aria-label="Anchor link for: memoization-add-a-memory" style="visibility: hidden;"></a> </h2> <p>Store results by input. On repeat calls, return the stored value instantly.</p> <pre data-lang="typescript" class="language-typescript z-code"><code class="language-typescript" data-lang="typescript"><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-storage z-type z-function z-ts">function</span> <span class="z-meta z-definition z-function z-ts"><span class="z-entity z-name z-function z-ts">memoSteps</span></span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">n</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-support z-type z-primitive z-ts">number</span></span><span class="z-punctuation z-separator z-parameter z-ts">,</span> <span class="z-variable z-parameter z-ts">cache</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">Map</span><span class="z-meta z-type z-parameters z-ts"><span class="z-punctuation z-definition z-typeparameters z-begin z-ts">&lt;</span></span><span class="z-meta z-type z-parameters z-ts"><span class="z-support z-type z-primitive z-ts">number</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-support z-type z-primitive z-ts">number</span></span><span class="z-meta z-type z-parameters z-ts"><span class="z-punctuation z-definition z-typeparameters z-end z-ts">&gt;</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-new z-expr z-ts"><span class="z-keyword z-operator z-new z-ts">new</span> <span class="z-entity z-name z-type z-ts">Map</span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span><span class="z-meta z-return z-type z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-support z-type z-primitive z-ts">number</span> </span><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">n</span> <span class="z-keyword z-operator z-relational z-ts">&lt;</span> <span class="z-constant z-numeric z-decimal z-ts">0</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-constant z-numeric z-decimal z-ts">0</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">n</span> <span class="z-keyword z-operator z-comparison z-ts">===</span> <span class="z-constant z-numeric z-decimal z-ts">0</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-constant z-numeric z-decimal z-ts">1</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">cache</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">has</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">n</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">cache</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">get</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">n</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-keyword z-operator z-logical z-ts">!</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">result</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-var z-expr z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">memoSteps</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">n</span> <span class="z-keyword z-operator z-arithmetic z-ts">-</span> <span class="z-constant z-numeric z-decimal z-ts">1</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-ts">cache</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-operator z-arithmetic z-ts">+</span></span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">memoSteps</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">n</span> <span class="z-keyword z-operator z-arithmetic z-ts">-</span> <span class="z-constant z-numeric z-decimal z-ts">3</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-ts">cache</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-operator z-arithmetic z-ts">+</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">memoSteps</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">n</span> <span class="z-keyword z-operator z-arithmetic z-ts">-</span> <span class="z-constant z-numeric z-decimal z-ts">5</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-ts">cache</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">cache</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">set</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">n</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-ts">result</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-variable z-other z-readwrite z-ts">result</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></code></pre> <p>Now <code>n = 35</code> returns instantly. Even <code>n = 100</code> is trivial because each distinct value of <code>n</code> is computed exactly once.</p> <h2 id="a-reusable-memoization-utility">A Reusable Memoization Utility<a class="zola-anchor" href="#a-reusable-memoization-utility" aria-label="Anchor link for: a-reusable-memoization-utility" style="visibility: hidden;"></a> </h2> <p>Rather than manually threading caches, build a generic memoizer:</p> <pre data-lang="typescript" class="language-typescript z-code"><code class="language-typescript" data-lang="typescript"><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-storage z-type z-function z-ts">function</span> <span class="z-meta z-definition z-function z-ts"><span class="z-entity z-name z-function z-ts">memoize</span></span><span class="z-meta z-type z-parameters z-ts"><span class="z-punctuation z-definition z-typeparameters z-begin z-ts">&lt;</span><span class="z-entity z-name z-type z-ts">T</span> <span class="z-storage z-modifier z-ts">extends</span> <span class="z-meta z-type z-function z-ts"><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-keyword z-operator z-rest z-ts">...</span><span class="z-variable z-parameter z-ts">args</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-support z-type z-primitive z-ts">any</span><span class="z-meta z-type z-tuple z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span></span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span></span> <span class="z-meta z-type z-function z-return z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> <span class="z-support z-type z-primitive z-ts">any</span></span><span class="z-punctuation z-definition z-typeparameters z-end z-ts">&gt;</span></span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">fn</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">T</span></span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span><span class="z-meta z-return z-type z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">T</span> </span><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">cache</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-new z-expr z-ts"><span class="z-keyword z-operator z-new z-ts">new</span> <span class="z-entity z-name z-type z-ts">Map</span><span class="z-meta z-type z-parameters z-ts"><span class="z-punctuation z-definition z-typeparameters z-begin z-ts">&lt;</span></span><span class="z-meta z-type z-parameters z-ts"><span class="z-support z-type z-primitive z-ts">string</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-entity z-name z-type z-ts">ReturnType</span><span class="z-meta z-type z-parameters z-ts"><span class="z-punctuation z-definition z-typeparameters z-begin z-ts">&lt;</span></span><span class="z-meta z-type z-parameters z-ts"><span class="z-entity z-name z-type z-ts">T</span></span><span class="z-meta z-type z-parameters z-ts"><span class="z-punctuation z-definition z-typeparameters z-end z-ts">&gt;</span></span></span><span class="z-meta z-type z-parameters z-ts"><span class="z-punctuation z-definition z-typeparameters z-end z-ts">&gt;</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-arrow z-ts"><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-keyword z-operator z-rest z-ts">...</span><span class="z-variable z-parameter z-ts">args</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">Parameters</span><span class="z-meta z-type z-parameters z-ts"><span class="z-punctuation z-definition z-typeparameters z-begin z-ts">&lt;</span></span><span class="z-meta z-type z-parameters z-ts"><span class="z-entity z-name z-type z-ts">T</span></span><span class="z-meta z-type z-parameters z-ts"><span class="z-punctuation z-definition z-typeparameters z-end z-ts">&gt;</span></span></span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span><span class="z-meta z-return z-type z-arrow z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">ReturnType</span><span class="z-meta z-type z-parameters z-ts"><span class="z-punctuation z-definition z-typeparameters z-begin z-ts">&lt;</span></span><span class="z-meta z-type z-parameters z-ts"><span class="z-entity z-name z-type z-ts">T</span></span><span class="z-meta z-type z-parameters z-ts"><span class="z-punctuation z-definition z-typeparameters z-end z-ts">&gt;</span></span> </span></span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">key</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-support z-constant z-json z-ts">JSON</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-json z-ts">stringify</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">args</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">cache</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">has</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">key</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">cache</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">get</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">key</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-keyword z-operator z-logical z-ts">!</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">result</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">fn</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-keyword z-operator z-spread z-ts">...</span><span class="z-variable z-other z-readwrite z-ts">args</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">cache</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">set</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">key</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-ts">result</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-variable z-other z-readwrite z-ts">result</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span></span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-control z-as z-ts">as</span> <span class="z-entity z-name z-type z-ts">T</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Usage</span> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">stepCount</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">memoize</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-arrow z-ts"><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">n</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-support z-type z-primitive z-ts">number</span></span><span class="z-punctuation z-separator z-parameter z-ts">,</span> <span class="z-variable z-parameter z-ts">steps</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-support z-type z-primitive z-ts">number</span><span class="z-meta z-type z-tuple z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span><span class="z-constant z-numeric z-decimal z-ts">1</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-constant z-numeric z-decimal z-ts">3</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-constant z-numeric z-decimal z-ts">5</span><span class="z-meta z-brace z-square z-ts">]</span></span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span><span class="z-meta z-return z-type z-arrow z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-support z-type z-primitive z-ts">number</span> </span></span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">n</span> <span class="z-keyword z-operator z-relational z-ts">&lt;</span> <span class="z-constant z-numeric z-decimal z-ts">0</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-constant z-numeric z-decimal z-ts">0</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">n</span> <span class="z-keyword z-operator z-comparison z-ts">===</span> <span class="z-constant z-numeric z-decimal z-ts">0</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-constant z-numeric z-decimal z-ts">1</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">steps</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">reduce</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-arrow z-ts"><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">sum</span><span class="z-punctuation z-separator z-parameter z-ts">,</span> <span class="z-variable z-parameter z-ts">step</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> </span><span class="z-variable z-other z-readwrite z-ts">sum</span> <span class="z-keyword z-operator z-arithmetic z-ts">+</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">stepCount</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">n</span> <span class="z-keyword z-operator z-arithmetic z-ts">-</span> <span class="z-variable z-other z-readwrite z-ts">step</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-ts">steps</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-constant z-numeric z-decimal z-ts">0</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></code></pre> <h2 id="lru-cache-bounded-memory">LRU Cache: Bounded Memory<a class="zola-anchor" href="#lru-cache-bounded-memory" aria-label="Anchor link for: lru-cache-bounded-memory" style="visibility: hidden;"></a> </h2> <p>For long-running applications, unbounded caches are dangerous. Implement a Least Recently Used cache:</p> <pre data-lang="typescript" class="language-typescript z-code"><code class="language-typescript" data-lang="typescript"><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-storage z-type z-class z-ts">class</span> <span class="z-entity z-name z-type z-class z-ts">LRUCache</span><span class="z-meta z-type z-parameters z-ts"><span class="z-punctuation z-definition z-typeparameters z-begin z-ts">&lt;</span><span class="z-entity z-name z-type z-ts">K</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-entity z-name z-type z-ts">V</span><span class="z-punctuation z-definition z-typeparameters z-end z-ts">&gt;</span></span> <span class="z-punctuation z-definition z-block z-ts">{</span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"> <span class="z-storage z-modifier z-ts">private</span><span class="z-meta z-field z-declaration z-ts"> <span class="z-meta z-definition z-property z-ts"><span class="z-variable z-object z-property z-ts">cache</span></span> <span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-new z-expr z-ts"><span class="z-keyword z-operator z-new z-ts">new</span> <span class="z-entity z-name z-type z-ts">Map</span><span class="z-meta z-type z-parameters z-ts"><span class="z-punctuation z-definition z-typeparameters z-begin z-ts">&lt;</span></span><span class="z-meta z-type z-parameters z-ts"><span class="z-entity z-name z-type z-ts">K</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-entity z-name z-type z-ts">V</span></span><span class="z-meta z-type z-parameters z-ts"><span class="z-punctuation z-definition z-typeparameters z-end z-ts">&gt;</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"> <span class="z-storage z-type z-ts">constructor</span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-storage z-modifier z-ts">private</span> <span class="z-variable z-parameter z-ts">maxSize</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-support z-type z-primitive z-ts">number</span></span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span><span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"> <span class="z-meta z-definition z-method z-ts"><span class="z-entity z-name z-function z-ts">get</span></span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">key</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">K</span></span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span><span class="z-meta z-return z-type z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">V</span> <span class="z-keyword z-operator z-type z-ts">|</span> <span class="z-support z-type z-builtin z-ts">undefined</span> </span><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-keyword z-operator z-logical z-ts">!</span><span class="z-meta z-function-call z-ts"><span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-object z-property z-ts">cache</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">has</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">key</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-constant z-language z-undefined z-ts">undefined</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-whitespace z-comment z-leading z-ts"> </span><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Move to end (most recently used)</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">value</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-object z-property z-ts">cache</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">get</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">key</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-keyword z-operator z-logical z-ts">!</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-object z-property z-ts">cache</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">delete</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">key</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-object z-property z-ts">cache</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">set</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">key</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-ts">value</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-variable z-other z-readwrite z-ts">value</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"> <span class="z-meta z-definition z-method z-ts"><span class="z-entity z-name z-function z-ts">set</span></span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">key</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">K</span></span><span class="z-punctuation z-separator z-parameter z-ts">,</span> <span class="z-variable z-parameter z-ts">value</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">V</span></span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span><span class="z-meta z-return z-type z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-support z-type z-primitive z-ts">void</span> </span><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-function-call z-ts"><span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-object z-property z-ts">cache</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">has</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">key</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-object z-property z-ts">cache</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">delete</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">key</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> <span class="z-keyword z-control z-conditional z-ts">else</span> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-object z-property z-ts">cache</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-dom z-ts">size</span> <span class="z-keyword z-operator z-relational z-ts">&gt;=</span> <span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">maxSize</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-whitespace z-comment z-leading z-ts"> </span><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Evict oldest (first) entry</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">firstKey</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-object z-property z-ts">cache</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">keys</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-function-call z-ts"><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">next</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-dom z-ts">value</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-object z-property z-ts">cache</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">delete</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">firstKey</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-object z-property z-ts">cache</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">set</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">key</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-ts">value</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"> <span class="z-meta z-definition z-method z-ts"><span class="z-entity z-name z-function z-ts">has</span></span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">key</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">K</span></span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span><span class="z-meta z-return z-type z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-support z-type z-primitive z-ts">boolean</span> </span><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-meta z-function-call z-ts"><span class="z-variable z-language z-this z-ts">this</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-object z-property z-ts">cache</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">has</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">key</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-meta z-method z-declaration z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-class z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span><span class="z-source z-ts"> </span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-storage z-type z-function z-ts">function</span> <span class="z-meta z-definition z-function z-ts"><span class="z-entity z-name z-function z-ts">memoizeWithLRU</span></span><span class="z-meta z-type z-parameters z-ts"><span class="z-punctuation z-definition z-typeparameters z-begin z-ts">&lt;</span><span class="z-entity z-name z-type z-ts">T</span> <span class="z-storage z-modifier z-ts">extends</span> <span class="z-meta z-type z-function z-ts"><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-keyword z-operator z-rest z-ts">...</span><span class="z-variable z-parameter z-ts">args</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-support z-type z-primitive z-ts">any</span><span class="z-meta z-type z-tuple z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span></span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span></span> <span class="z-meta z-type z-function z-return z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> <span class="z-support z-type z-primitive z-ts">any</span></span><span class="z-punctuation z-definition z-typeparameters z-end z-ts">&gt;</span></span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-parameters z-ts"> <span class="z-variable z-parameter z-ts">fn</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">T</span></span><span class="z-punctuation z-separator z-parameter z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-parameters z-ts"> <span class="z-variable z-parameter z-ts">maxSize</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-support z-type z-primitive z-ts">number</span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-constant z-numeric z-decimal z-ts">1000</span><span class="z-punctuation z-separator z-parameter z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span><span class="z-meta z-return z-type z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">T</span> </span><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">cache</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-new z-expr z-ts"><span class="z-keyword z-operator z-new z-ts">new</span> <span class="z-entity z-name z-type z-ts">LRUCache</span><span class="z-meta z-type z-parameters z-ts"><span class="z-punctuation z-definition z-typeparameters z-begin z-ts">&lt;</span></span><span class="z-meta z-type z-parameters z-ts"><span class="z-support z-type z-primitive z-ts">string</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-entity z-name z-type z-ts">ReturnType</span><span class="z-meta z-type z-parameters z-ts"><span class="z-punctuation z-definition z-typeparameters z-begin z-ts">&lt;</span></span><span class="z-meta z-type z-parameters z-ts"><span class="z-entity z-name z-type z-ts">T</span></span><span class="z-meta z-type z-parameters z-ts"><span class="z-punctuation z-definition z-typeparameters z-end z-ts">&gt;</span></span></span><span class="z-meta z-type z-parameters z-ts"><span class="z-punctuation z-definition z-typeparameters z-end z-ts">&gt;</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">maxSize</span><span class="z-meta z-brace z-round z-ts">)</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-arrow z-ts"><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-keyword z-operator z-rest z-ts">...</span><span class="z-variable z-parameter z-ts">args</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">Parameters</span><span class="z-meta z-type z-parameters z-ts"><span class="z-punctuation z-definition z-typeparameters z-begin z-ts">&lt;</span></span><span class="z-meta z-type z-parameters z-ts"><span class="z-entity z-name z-type z-ts">T</span></span><span class="z-meta z-type z-parameters z-ts"><span class="z-punctuation z-definition z-typeparameters z-end z-ts">&gt;</span></span></span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span><span class="z-meta z-return z-type z-arrow z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-entity z-name z-type z-ts">ReturnType</span><span class="z-meta z-type z-parameters z-ts"><span class="z-punctuation z-definition z-typeparameters z-begin z-ts">&lt;</span></span><span class="z-meta z-type z-parameters z-ts"><span class="z-entity z-name z-type z-ts">T</span></span><span class="z-meta z-type z-parameters z-ts"><span class="z-punctuation z-definition z-typeparameters z-end z-ts">&gt;</span></span> </span></span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">key</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-support z-constant z-json z-ts">JSON</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-json z-ts">stringify</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">args</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">cache</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">has</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">key</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">cache</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">get</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">key</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-keyword z-operator z-logical z-ts">!</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">result</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">fn</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-keyword z-operator z-spread z-ts">...</span><span class="z-variable z-other z-readwrite z-ts">args</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">cache</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">set</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">key</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-variable z-other z-readwrite z-ts">result</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-variable z-other z-readwrite z-ts">result</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span></span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-keyword z-control z-as z-ts">as</span> <span class="z-entity z-name z-type z-ts">T</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></code></pre> <h2 id="why-it-works">Why It Works<a class="zola-anchor" href="#why-it-works" aria-label="Anchor link for: why-it-works" style="visibility: hidden;"></a> </h2> <p>Memoization transforms a branching recursion tree into a directed acyclic graph. Each unique input becomes a single node. Compute once, reuse forever.</p> <p>It's like saving a phone number in your contacts. The first time someone gives you their number, you type it in. Every time after that, you just look it up instead of asking again. Memoization is your contact list for function results.</p> <h2 id="when-to-reach-for-memoization">When to Reach for Memoization<a class="zola-anchor" href="#when-to-reach-for-memoization" aria-label="Anchor link for: when-to-reach-for-memoization" style="visibility: hidden;"></a> </h2> <p><strong>It shines when you have:</strong></p> <ol> <li><strong>Pure functions</strong> — output depends only on input, no side effects</li> <li><strong>Overlapping subproblems</strong> — the same inputs recur across different execution paths</li> <li><strong>Manageable state space</strong> — unique inputs fit comfortably in memory</li> </ol> <p><strong>Common applications:</strong></p> <ul> <li><strong>Dynamic programming</strong>: Fibonacci, coin change, edit distance, longest common subsequence</li> <li><strong>Graph algorithms</strong>: path counting, shortest paths with repeated states</li> <li><strong>Game engines</strong>: transposition tables in chess cache board evaluations to skip re-analyzing identical positions</li> <li><strong>API responses</strong>: cache expensive computations or external calls</li> </ul> <h2 id="the-bottom-up-alternative">The Bottom-Up Alternative<a class="zola-anchor" href="#the-bottom-up-alternative" aria-label="Anchor link for: the-bottom-up-alternative" style="visibility: hidden;"></a> </h2> <p>Sometimes you can flip the script—compute iteratively from base cases up:</p> <pre data-lang="typescript" class="language-typescript z-code"><code class="language-typescript" data-lang="typescript"><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-storage z-type z-function z-ts">function</span> <span class="z-meta z-definition z-function z-ts"><span class="z-entity z-name z-function z-ts">stepCountDP</span></span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">n</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-support z-type z-primitive z-ts">number</span></span><span class="z-punctuation z-separator z-parameter z-ts">,</span> <span class="z-variable z-parameter z-ts">steps</span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-support z-type z-primitive z-ts">number</span><span class="z-meta z-type z-tuple z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span><span class="z-meta z-array z-literal z-ts"> <span class="z-meta z-brace z-square z-ts">[</span><span class="z-constant z-numeric z-decimal z-ts">1</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-constant z-numeric z-decimal z-ts">3</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-constant z-numeric z-decimal z-ts">5</span><span class="z-meta z-brace z-square z-ts">]</span></span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span><span class="z-meta z-return z-type z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-support z-type z-primitive z-ts">number</span> </span><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">dp</span></span><span class="z-meta z-type z-annotation z-ts"><span class="z-keyword z-operator z-type z-annotation z-ts">:</span> <span class="z-support z-type z-primitive z-ts">number</span><span class="z-meta z-type z-tuple z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-meta z-brace z-square z-ts">]</span></span> </span></span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-new z-expr z-ts"><span class="z-keyword z-operator z-new z-ts">new</span> <span class="z-entity z-name z-type z-ts">Array</span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">n</span> <span class="z-keyword z-operator z-arithmetic z-ts">+</span> <span class="z-constant z-numeric z-decimal z-ts">1</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-meta z-function-call z-ts"><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">fill</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-constant z-numeric z-decimal z-ts">0</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-other z-readwrite z-ts">dp</span><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-constant z-numeric z-decimal z-ts">0</span><span class="z-meta z-brace z-square z-ts">]</span></span> <span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-constant z-numeric z-decimal z-ts">1</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-loop z-ts">for</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">let</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-readwrite z-ts">i</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-constant z-numeric z-decimal z-ts">1</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> <span class="z-variable z-other z-readwrite z-ts">i</span> <span class="z-keyword z-operator z-relational z-ts">&lt;=</span> <span class="z-variable z-other z-readwrite z-ts">n</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> <span class="z-variable z-other z-readwrite z-ts">i</span><span class="z-keyword z-operator z-increment z-ts">++</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-loop z-ts">for</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">step</span></span> </span></span><span class="z-keyword z-operator z-expression z-of z-ts">of</span> <span class="z-variable z-other z-readwrite z-ts">steps</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">i</span> <span class="z-keyword z-operator z-arithmetic z-ts">-</span> <span class="z-variable z-other z-readwrite z-ts">step</span> <span class="z-keyword z-operator z-relational z-ts">&gt;=</span> <span class="z-constant z-numeric z-decimal z-ts">0</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-other z-readwrite z-ts">dp</span><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-variable z-other z-readwrite z-ts">i</span><span class="z-meta z-brace z-square z-ts">]</span></span> <span class="z-keyword z-operator z-assignment z-compound z-ts">+=</span> <span class="z-variable z-other z-readwrite z-ts">dp</span><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-variable z-other z-readwrite z-ts">i</span> <span class="z-keyword z-operator z-arithmetic z-ts">-</span> <span class="z-variable z-other z-readwrite z-ts">step</span><span class="z-meta z-brace z-square z-ts">]</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-variable z-other z-readwrite z-ts">dp</span><span class="z-meta z-array z-literal z-ts"><span class="z-meta z-brace z-square z-ts">[</span><span class="z-variable z-other z-readwrite z-ts">n</span><span class="z-meta z-brace z-square z-ts">]</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></code></pre> <ul> <li><strong>Time</strong>: O(n × number of steps)</li> <li><strong>Space</strong>: O(n)</li> <li><strong>No recursion</strong>: avoids stack overflow for large <code>n</code></li> </ul> <h2 id="choosing-your-approach">Choosing Your Approach<a class="zola-anchor" href="#choosing-your-approach" aria-label="Anchor link for: choosing-your-approach" style="visibility: hidden;"></a> </h2> <table><thead><tr><th>Approach</th><th>Best For</th></tr></thead><tbody> <tr><td><strong>Memoization</strong></td><td>Natural recursive problems, lazy evaluation, when not all subproblems are needed</td></tr> <tr><td><strong>Bottom-up DP</strong></td><td>Clear state space, tight memory control, avoiding recursion limits</td></tr> <tr><td><strong>LRU caching</strong></td><td>Long-running services, memory-constrained environments</td></tr> </tbody></table> <h2 id="practical-gotchas">Practical Gotchas<a class="zola-anchor" href="#practical-gotchas" aria-label="Anchor link for: practical-gotchas" style="visibility: hidden;"></a> </h2> <ol> <li> <p><strong>Memory growth</strong>: Unbounded caches can leak memory. Use LRU or clear caches periodically.</p> </li> <li> <p><strong>Cache key design</strong>: Keys must be immutable and capture all inputs that affect the result. Use <code>JSON.stringify</code> for simplicity, but beware of object key ordering.</p> </li> <li> <p><strong>Break-even point</strong>: For tiny inputs, cache overhead can exceed computation cost. Profile before optimizing.</p> </li> <li> <p><strong>Cache invalidation</strong>: If the underlying behavior changes (config updates, time-dependent logic), version your cache or clear it.</p> </li> </ol> <h2 id="the-takeaway">The Takeaway<a class="zola-anchor" href="#the-takeaway" aria-label="Anchor link for: the-takeaway" style="visibility: hidden;"></a> </h2> <p>Memoization is a small change with outsized impact. Cache results by input, reuse them on repeat calls. For problems with overlapping subproblems, it transforms exponential blowups into linear-time solutions.</p> <p>Design your keys carefully, bound your memory when needed, and you'll have code that's both elegant and fast.</p> 2025-12-10T00:00:00+00:00 2025-12-10T00:00:00+00:00 https://rajnandan.com/posts/enterprise-ai-2025-37-billion/ <p>Enterprise AI has grown from $1.7 billion in 2023 to $37 billion in 2025. That's not a typo. In just three years, AI has captured over 6% of the global SaaS market and become the fastest-scaling software category in history.</p> <p>The bubble fears haven't materialized. The demand side tells a different story: broad adoption, real revenue, and productivity gains at scale.</p> <h2 id="where-the-money-goes">Where the Money Goes<a class="zola-anchor" href="#where-the-money-goes" aria-label="Anchor link for: where-the-money-goes" style="visibility: hidden;"></a> </h2> <p>The $37 billion breaks down into two main buckets:</p> <p><strong>Applications ($19 billion):</strong> The user-facing products that leverage AI models</p> <ul> <li>Departmental AI: $7.3B (coding, IT, marketing, customer success)</li> <li>Horizontal AI: $8.4B (copilots, agents, productivity tools)</li> <li>Vertical AI: $3.5B (healthcare, legal, finance)</li> </ul> <p><strong>Infrastructure ($18 billion):</strong> The picks and shovels</p> <ul> <li>Foundation model APIs: $12.5B</li> <li>Model training infrastructure: $4B</li> <li>AI infrastructure (storage, retrieval, orchestration): $1.5B</li> </ul> <p>The shift is notable: more than half of enterprise AI spend now goes to applications rather than infrastructure. Enterprises are prioritizing immediate productivity gains over long-term infrastructure bets.</p> <h2 id="the-buy-vs-build-flip">The Buy vs. Build Flip<a class="zola-anchor" href="#the-buy-vs-build-flip" aria-label="Anchor link for: the-buy-vs-build-flip" style="visibility: hidden;"></a> </h2> <p>In 2024, enterprises were split - 47% built internally, 53% purchased. A year later, it's not even close: <strong>76% of AI use cases are now purchased rather than built internally</strong>.</p> <p>The conventional wisdom that enterprises would build most AI solutions themselves didn't hold. Ready-made AI solutions are reaching production faster and demonstrating immediate value while internal builds mature.</p> <h2 id="ai-buyers-are-different">AI Buyers Are Different<a class="zola-anchor" href="#ai-buyers-are-different" aria-label="Anchor link for: ai-buyers-are-different" style="visibility: hidden;"></a> </h2> <p>Enterprise AI buyers behave differently than traditional software buyers:</p> <ul> <li><strong>47% conversion rate</strong> for AI deals (vs. 25% for traditional SaaS)</li> <li><strong>27% of AI spend</strong> comes through product-led growth (vs. 7% for traditional software)</li> <li>Organizations surface 10+ potential use cases but focus on near-term productivity gains</li> </ul> <p>That 47% conversion rate is remarkable. Once an organization commits to exploring an AI solution, deals close at nearly twice the rate of traditional software. The value is immediate and clear.</p> <h2 id="plg-is-eating-enterprise-ai">PLG Is Eating Enterprise AI<a class="zola-anchor" href="#plg-is-eating-enterprise-ai" aria-label="Anchor link for: plg-is-eating-enterprise-ai" style="visibility: hidden;"></a> </h2> <p>Product-led growth is transforming enterprise AI adoption. Individual users now drive AI adoption at 4x the rate of traditional software.</p> <p>The examples are striking:</p> <ul> <li><strong>Cursor</strong> reached $200 million in revenue before hiring a single enterprise sales rep</li> <li><strong>n8n</strong> formalized contracts only after hundreds of employees were already active users</li> <li><strong>ElevenLabs</strong>, <strong>Gamma</strong>, and others scaled the same way</li> </ul> <p>When you account for "shadow AI adoption" - employees using personal credit cards for tools like ChatGPT Plus - PLG-driven tools may represent close to 40% of application AI spend.</p> <p>The pattern: developers and technical teams discover tools for individual use, prove value in day-to-day work, and create bottom-up demand that converts to enterprise contracts.</p> <h2 id="startups-are-winning-the-application-layer">Startups Are Winning the Application Layer<a class="zola-anchor" href="#startups-are-winning-the-application-layer" aria-label="Anchor link for: startups-are-winning-the-application-layer" style="visibility: hidden;"></a> </h2> <p>At the AI application layer, startups now capture nearly <strong>$2 in revenue for every $1 earned by incumbents</strong> - 63% of the market, up from 36% last year.</p> <p>This shouldn't be happening. Incumbents have distribution, data moats, enterprise relationships, sales teams, and massive balance sheets. Yet AI-native startups are out-executing across the fastest-growing categories.</p> <p><strong>Why startups win:</strong></p> <table><thead><tr><th>Department</th><th>Startup Share</th><th>Why</th></tr></thead><tbody> <tr><td>Product + Engineering</td><td>71%</td><td>Cursor beat GitHub Copilot by shipping faster - repo-level context, multi-file editing, model-agnostic approach</td></tr> <tr><td>Sales</td><td>78%</td><td>Clay, Actively attack workflows Salesforce doesn't own - research, personalization, enrichment from unstructured signals</td></tr> <tr><td>Finance + Operations</td><td>91%</td><td>Incumbents can't move fast enough; startups like Rillet and Numeric build AI-first ERPs</td></tr> </tbody></table> <p>The story changes at the infrastructure layer, where incumbents hold 56% of the market. Databricks, Snowflake, MongoDB, and Datadog have seen meaningful re-acceleration as even AI-native builders choose existing platforms.</p> <h2 id="coding-the-first-killer-use-case">Coding: The First Killer Use Case<a class="zola-anchor" href="#coding-the-first-killer-use-case" aria-label="Anchor link for: coding-the-first-killer-use-case" style="visibility: hidden;"></a> </h2> <p>Coding is the breakout category. At $4 billion, it represents 55% of all departmental AI spend - the largest category across the entire application layer.</p> <p>The numbers:</p> <ul> <li>50% of developers now use AI coding tools daily (65% in top-quartile orgs)</li> <li>Code completion alone is $2.3B</li> <li>Code agents and AI app builders exploded from near-zero</li> <li>Teams report 15%+ velocity gains</li> </ul> <p>AI now touches the entire software development lifecycle: prototyping (Lovable), code refactoring (OpenHands), design-to-code (Weaver), QA (Meticulous), PRs (Graphite), SRE (Resolve), and deployment (Harness).</p> <h2 id="anthropic-is-winning">Anthropic Is Winning<a class="zola-anchor" href="#anthropic-is-winning" aria-label="Anchor link for: anthropic-is-winning" style="visibility: hidden;"></a> </h2> <p>The foundation model landscape shifted decisively. Anthropic now earns an estimated <strong>40% of enterprise LLM spend</strong>, up from 24% last year and 12% in 2023. OpenAI fell to 27% from 50% in 2023.</p> <p>The three leaders - Anthropic, OpenAI, and Google - account for 88% of enterprise LLM API usage.</p> <p>Anthropic's dominance comes from coding. They command <strong>54% of the coding market</strong> (vs. 21% for OpenAI), driven by an 18-month run atop leaderboards starting with Claude Sonnet 3.5 in June 2024.</p> <h2 id="agents-mostly-hype-for-now">Agents: Mostly Hype (For Now)<a class="zola-anchor" href="#agents-mostly-hype-for-now" aria-label="Anchor link for: agents-mostly-hype-for-now" style="visibility: hidden;"></a> </h2> <p>For all the talk of agents, real production architectures remain surprisingly simple:</p> <ul> <li>Only <strong>16% of enterprise</strong> and <strong>27% of startup</strong> deployments qualify as true agents</li> <li>Most are still fixed-sequence or routing-based workflows around a single model call</li> <li>Prompt design remains the dominant customization technique, followed by RAG</li> <li>Fine-tuning, tool calling, and RL are still niche</li> </ul> <p>Copilots dominate horizontal AI with 86% share ($7.2B). Agent platforms capture only 10% ($750M).</p> <p>Strip away the hype and most "AI agents" are basic if-then logic around a model call. The architecture works for today's use cases - but it reveals how early we are.</p> <h2 id="healthcare-leads-vertical-ai">Healthcare Leads Vertical AI<a class="zola-anchor" href="#healthcare-leads-vertical-ai" aria-label="Anchor link for: healthcare-leads-vertical-ai" style="visibility: hidden;"></a> </h2> <p>Vertical AI solutions hit $3.5 billion, nearly 3x the prior year. Healthcare alone captures 43% - approximately $1.5 billion.</p> <p>The bulk concentrates in administrative and clinical-adjacent workflows:</p> <ul> <li><strong>Ambient scribes</strong>: $600M (+2.4x YoY), minting two new unicorns (Abridge, Ambience)</li> <li>Clinicians spend one hour documenting for every five hours of care</li> <li>Scribes that reduce documentation time by 50%+ are transformational</li> </ul> <p>Beyond healthcare: legal ($650M), creator tools ($360M), and government ($350M). Adoption is strongest in industries historically underserved by software - fields defined by manual, unstructured workflows that once depended on human services.</p> <h2 id="what-s-next">What's Next<a class="zola-anchor" href="#what-s-next" aria-label="Anchor link for: what-s-next" style="visibility: hidden;"></a> </h2> <p>Five predictions for 2026:</p> <ol> <li> <p><strong>AI will exceed human performance in daily practical programming tasks.</strong> The best models keep getting better in verifiable domains like math and code.</p> </li> <li> <p><strong>Jevons' paradox continues.</strong> Net spend rises despite falling inference costs, driven by orders-of-magnitude increases in inference volume.</p> </li> <li> <p><strong>Explainability and governance go mainstream.</strong> As agents gain autonomy, the ability to explain and govern their decisions becomes critical. Governments will ask for audit logs.</p> </li> <li> <p><strong>Models move to the edge.</strong> Low-latency requirements, privacy, and cost push compute on-device. Mobile manufacturers ship dedicated low-power GPU compute.</p> </li> <li> <p><strong>Benchmarks saturate but fail to capture real-world efficacy.</strong> For frontier use cases like coding, users are price-insensitive and will pay more for performance.</p> </li> </ol> <h2 id="the-takeaway">The Takeaway<a class="zola-anchor" href="#the-takeaway" aria-label="Anchor link for: the-takeaway" style="visibility: hidden;"></a> </h2> <p>Enterprise AI is no longer speculative. It's a $37 billion market growing faster than any software category in history.</p> <p>The dynamics are clear:</p> <ul> <li>Enterprises are buying, not building</li> <li>Startups are winning the application layer</li> <li>Coding is the first killer use case</li> <li>Anthropic leads the enterprise LLM market</li> <li>Agents are mostly hype - for now</li> <li>PLG is driving adoption at unprecedented rates</li> </ul> <p>Three years in, the transformation is real. The value is clear. And the pace isn't slowing.</p> 2025-12-08T00:00:00+00:00 2025-12-08T00:00:00+00:00 https://rajnandan.com/posts/mckinsey-state-of-ai-2025-analysis/ <p>McKinsey just dropped their <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai">State of AI 2025 report</a>, surveying nearly 2,000 participants across 105 countries. The headline: almost everyone is using AI now, but almost no one is seeing meaningful enterprise-wide impact.</p> <p>Three years into the generative AI era, this is the defining tension.</p> <h2 id="the-adoption-impact-gap">The Adoption-Impact Gap<a class="zola-anchor" href="#the-adoption-impact-gap" aria-label="Anchor link for: the-adoption-impact-gap" style="visibility: hidden;"></a> </h2> <p>The numbers tell a clear story:</p> <ul> <li><strong>88%</strong> of organizations are using AI in at least one business function (up from 78% last year)</li> <li><strong>62%</strong> are experimenting with AI agents</li> <li>But only <strong>39%</strong> report any EBIT impact at the enterprise level</li> <li>And most of those say AI accounts for less than <strong>5%</strong> of EBIT</li> </ul> <p>Nearly nine in ten companies are using AI. Fewer than four in ten see it on their bottom line.</p> <p>This isn't a technology problem. It's a deployment problem.</p> <h2 id="most-organizations-are-still-piloting">Most Organizations Are Still Piloting<a class="zola-anchor" href="#most-organizations-are-still-piloting" aria-label="Anchor link for: most-organizations-are-still-piloting" style="visibility: hidden;"></a> </h2> <p>The report's key finding: <strong>two-thirds of organizations have not begun scaling AI across the enterprise</strong>. They're stuck in experimentation and pilot phases.</p> <p>Larger companies are ahead—nearly half of organizations with $5B+ in revenue have reached the scaling phase. But for companies under $100M in revenue, it's only 29%.</p> <p>The pattern is familiar from every previous wave of enterprise technology: early adopters move fast, everyone else gets stuck in "proof of concept" purgatory.</p> <h2 id="ai-agents-high-curiosity-low-scale">AI Agents: High Curiosity, Low Scale<a class="zola-anchor" href="#ai-agents-high-curiosity-low-scale" aria-label="Anchor link for: ai-agents-high-curiosity-low-scale" style="visibility: hidden;"></a> </h2> <p>Agents are the hot topic, and the numbers reflect it:</p> <ul> <li><strong>62%</strong> are at least experimenting with AI agents</li> <li><strong>23%</strong> are scaling agentic AI somewhere in the enterprise</li> <li>But in any given function, <strong>no more than 10%</strong> are scaling agents</li> </ul> <p>Agent adoption is highest in:</p> <ul> <li><strong>IT and knowledge management</strong> (service-desk automation, deep research)</li> <li><strong>Tech, media, and telecom</strong> industries</li> <li><strong>Healthcare</strong></li> </ul> <p>The pattern here is revealing: agents work best where the use case is narrow, repeatable, and the cost of errors is manageable. IT service desks. Document retrieval. Scheduling. The more open-ended the task, the harder it is to scale.</p> <h2 id="what-high-performers-do-differently">What High Performers Do Differently<a class="zola-anchor" href="#what-high-performers-do-differently" aria-label="Anchor link for: what-high-performers-do-differently" style="visibility: hidden;"></a> </h2> <p>McKinsey defines "AI high performers" as those who attribute 5%+ of EBIT to AI and report "significant" value. They represent about 6% of respondents.</p> <p>What separates them:</p> <h3 id="1-they-aim-for-transformation-not-just-efficiency">1. They Aim for Transformation, Not Just Efficiency<a class="zola-anchor" href="#1-they-aim-for-transformation-not-just-efficiency" aria-label="Anchor link for: 1-they-aim-for-transformation-not-just-efficiency" style="visibility: hidden;"></a> </h3> <p>80% of all organizations set efficiency as an AI objective. That's table stakes.</p> <p>High performers are <strong>3x more likely</strong> to say they intend to use AI for <strong>transformative change</strong>—not just cost cutting.</p> <p>Companies that set growth and innovation as objectives (not just efficiency) report better outcomes across customer satisfaction, competitive differentiation, profitability, and revenue growth.</p> <h3 id="2-they-redesign-workflows">2. They Redesign Workflows<a class="zola-anchor" href="#2-they-redesign-workflows" aria-label="Anchor link for: 2-they-redesign-workflows" style="visibility: hidden;"></a> </h3> <p>This is one of the strongest predictors of impact. High performers are <strong>nearly 3x more likely</strong> to have fundamentally redesigned workflows when deploying AI.</p> <p>Most organizations bolt AI onto existing processes. High performers rebuild the process around AI capabilities.</p> <h3 id="3-they-scale-faster-and-wider">3. They Scale Faster and Wider<a class="zola-anchor" href="#3-they-scale-faster-and-wider" aria-label="Anchor link for: 3-they-scale-faster-and-wider" style="visibility: hidden;"></a> </h3> <p>High performers use AI in more functions. They're also 3x more likely to be scaling AI agents (not just piloting them).</p> <p>Three-quarters of high performers have scaled or are scaling AI, compared to one-third of everyone else.</p> <h3 id="4-leadership-owns-it">4. Leadership Owns It<a class="zola-anchor" href="#4-leadership-owns-it" aria-label="Anchor link for: 4-leadership-owns-it" style="visibility: hidden;"></a> </h3> <p>High performers are <strong>3x more likely</strong> to strongly agree that senior leaders demonstrate ownership and commitment to AI initiatives.</p> <p>Leaders at these organizations aren't just sponsors—they're role-modeling AI use.</p> <h3 id="5-they-invest-more">5. They Invest More<a class="zola-anchor" href="#5-they-invest-more" aria-label="Anchor link for: 5-they-invest-more" style="visibility: hidden;"></a> </h3> <p>More than one-third of high performers spend over 20% of their digital budgets on AI technologies.</p> <p>You can't pilot your way to transformation.</p> <h2 id="the-workforce-question">The Workforce Question<a class="zola-anchor" href="#the-workforce-question" aria-label="Anchor link for: the-workforce-question" style="visibility: hidden;"></a> </h2> <p>Expectations are mixed:</p> <ul> <li><strong>43%</strong> expect no change in workforce size due to AI</li> <li><strong>32%</strong> expect workforce reductions of 3%+ in the next year</li> <li><strong>13%</strong> expect increases of 3%+</li> </ul> <p>What's interesting: larger organizations are more likely to expect reductions. And AI high performers are more likely to expect <em>change</em>—in either direction.</p> <p>Meanwhile, most organizations are <strong>hiring for AI roles</strong>—particularly software engineers and data engineers. Even as some functions shrink, the demand for AI talent is growing.</p> <h2 id="risk-mitigation-is-improving-slowly">Risk Mitigation Is Improving (Slowly)<a class="zola-anchor" href="#risk-mitigation-is-improving-slowly" aria-label="Anchor link for: risk-mitigation-is-improving-slowly" style="visibility: hidden;"></a> </h2> <p>The good news: organizations are starting to take AI risks seriously.</p> <ul> <li>In 2022, organizations were mitigating an average of <strong>2 AI-related risks</strong></li> <li>In 2025, that number is <strong>4</strong></li> </ul> <p>The most common consequences experienced:</p> <ul> <li><strong>Inaccuracy</strong> (31% of respondents)</li> <li>Followed by explainability issues, privacy concerns, and regulatory compliance</li> </ul> <p>High performers—who have deployed more use cases—are more likely to experience negative consequences, particularly around IP infringement and regulatory compliance. But they're also more likely to have mitigation measures in place.</p> <h2 id="what-this-means">What This Means<a class="zola-anchor" href="#what-this-means" aria-label="Anchor link for: what-this-means" style="visibility: hidden;"></a> </h2> <p>Three takeaways:</p> <h3 id="1-the-window-for-competitive-advantage-is-open">1. The Window for Competitive Advantage Is Open<a class="zola-anchor" href="#1-the-window-for-competitive-advantage-is-open" aria-label="Anchor link for: 1-the-window-for-competitive-advantage-is-open" style="visibility: hidden;"></a> </h3> <p>Most organizations are stuck in pilots. The ones moving to production-scale deployment have a real opportunity to pull ahead.</p> <h3 id="2-efficiency-isn-t-enough">2. Efficiency Isn't Enough<a class="zola-anchor" href="#2-efficiency-isn-t-enough" aria-label="Anchor link for: 2-efficiency-isn-t-enough" style="visibility: hidden;"></a> </h3> <p>Cost reduction is the default goal. But the organizations seeing the most value are using AI to drive growth and innovation—not just to cut headcount.</p> <h3 id="3-workflow-redesign-is-the-unlock">3. Workflow Redesign Is the Unlock<a class="zola-anchor" href="#3-workflow-redesign-is-the-unlock" aria-label="Anchor link for: 3-workflow-redesign-is-the-unlock" style="visibility: hidden;"></a> </h3> <p>Bolting AI onto existing processes produces marginal gains. Redesigning workflows around AI capabilities produces transformational outcomes.</p> <p>This is the hardest part. It requires not just technology investment, but organizational change, process engineering, and leadership commitment.</p> <hr /> <p>The State of AI 2025 confirms what many of us see in practice: AI is everywhere, but enterprise value is still rare. The technology works. The challenge now is organizational.</p> <p>The gap between "we use AI" and "AI transformed our business" is where the real work happens.</p> 2025-12-06T00:00:00+00:00 2025-12-06T00:00:00+00:00 https://rajnandan.com/posts/introducing-paperpin-ai-surveys/ <p>I've been working on a new project called <a rel="noopener nofollow noreferrer" target="_blank" href="https://paperpin.ai">PaperPin</a>—an AI-powered survey platform that uses synthetic respondents to deliver research insights fast.</p> <h2 id="the-problem-with-traditional-surveys">The Problem with Traditional Surveys<a class="zola-anchor" href="#the-problem-with-traditional-surveys" aria-label="Anchor link for: the-problem-with-traditional-surveys" style="visibility: hidden;"></a> </h2> <p>If you've ever run user research, you know the pain:</p> <ul> <li><strong>Recruiting takes weeks.</strong> Finding the right respondents, screening them, scheduling—it's a logistics nightmare.</li> <li><strong>Panels are expensive.</strong> Quality respondents cost money, and costs scale linearly with sample size.</li> <li><strong>Turnaround is slow.</strong> By the time you get results, the product has already shipped or the decision has been made.</li> <li><strong>Niche audiences are nearly impossible.</strong> Good luck finding 50 CFOs of mid-market SaaS companies who will take your survey.</li> </ul> <p>Traditional survey panels work, but they're slow, expensive, and don't scale well for rapid iteration.</p> <h2 id="what-paperpin-does-differently">What PaperPin Does Differently<a class="zola-anchor" href="#what-paperpin-does-differently" aria-label="Anchor link for: what-paperpin-does-differently" style="visibility: hidden;"></a> </h2> <p>PaperPin replaces human panels with AI-generated synthetic respondents. Here's how it works:</p> <h3 id="1-define-your-persona">1. Define Your Persona<a class="zola-anchor" href="#1-define-your-persona" aria-label="Anchor link for: 1-define-your-persona" style="visibility: hidden;"></a> </h3> <p>Specify the demographics, behaviors, and psychographics of your target audience. Want responses from "35-45 year old product managers at enterprise SaaS companies who prioritize developer experience"? Done.</p> <h3 id="2-ai-simulates-authentic-responses">2. AI Simulates Authentic Responses<a class="zola-anchor" href="#2-ai-simulates-authentic-responses" aria-label="Anchor link for: 2-ai-simulates-authentic-responses" style="visibility: hidden;"></a> </h3> <p>Advanced LLMs embody your persona with nuanced, contextual understanding. They respond authentically, capturing the voice and perspective of real people in that demographic.</p> <h3 id="3-get-results-in-minutes">3. Get Results in Minutes<a class="zola-anchor" href="#3-get-results-in-minutes" aria-label="Anchor link for: 3-get-results-in-minutes" style="visibility: hidden;"></a> </h3> <p>What once took weeks now takes minutes. No recruitment delays, no scheduling, no no-shows.</p> <h3 id="4-validated-accuracy">4. Validated Accuracy<a class="zola-anchor" href="#4-validated-accuracy" aria-label="Anchor link for: 4-validated-accuracy" style="visibility: hidden;"></a> </h3> <p>Results are calibrated against real human data. PaperPin achieves <strong>95% correlation</strong> with traditional human panel surveys across thousands of validation studies.</p> <h2 id="what-you-can-research">What You Can Research<a class="zola-anchor" href="#what-you-can-research" aria-label="Anchor link for: what-you-can-research" style="visibility: hidden;"></a> </h2> <p>PaperPin supports multiple research campaign types:</p> <ul> <li><strong>Concept &amp; Product Testing</strong> — Test new product ideas, measure purchase intent, validate product-market fit</li> <li><strong>Brand &amp; Market Research</strong> — Measure brand awareness, evaluate positioning, analyze competitors</li> <li><strong>Customer Insights</strong> — Track CSAT, measure NPS, identify what drives customer advocacy</li> <li><strong>UX Testing</strong> — Gather detailed product feedback, evaluate usability, prioritize features</li> <li><strong>Ad &amp; Message Testing</strong> — Measure ad effectiveness, test messaging variants, compare value propositions</li> <li><strong>Pricing Research</strong> — Understand price sensitivity, estimate willingness to pay, optimize pricing strategy</li> </ul> <h2 id="why-synthetic-respondents">Why Synthetic Respondents?<a class="zola-anchor" href="#why-synthetic-respondents" aria-label="Anchor link for: why-synthetic-respondents" style="visibility: hidden;"></a> </h2> <p>I built PaperPin because I kept running into the same problem: I needed quick feedback on product decisions, but traditional research was too slow and expensive for the pace of iteration I wanted.</p> <p>Synthetic respondents aren't a replacement for all human research. They're a tool for:</p> <ul> <li><strong>Early-stage validation</strong> when you need directional insights fast</li> <li><strong>Rapid iteration</strong> when you're testing multiple variants</li> <li><strong>Niche audiences</strong> that are hard or expensive to recruit</li> <li><strong>Budget-conscious research</strong> when panel costs don't make sense</li> </ul> <p>The goal isn't perfect fidelity—it's getting useful signal quickly so you can make better decisions faster.</p> <h2 id="how-it-works-under-the-hood">How It Works Under the Hood<a class="zola-anchor" href="#how-it-works-under-the-hood" aria-label="Anchor link for: how-it-works-under-the-hood" style="visibility: hidden;"></a> </h2> <p>Each synthetic respondent is generated based on your cohort definition. The AI doesn't just answer questions randomly—it maintains consistent persona characteristics across the survey, providing both Likert-scale ratings and qualitative explanations.</p> <p>For example, if you're testing a new pricing page:</p> <ol> <li>Define a cohort: "Small business owners, 5-20 employees, using spreadsheets for invoicing"</li> <li>Create your survey with questions about clarity, trust signals, willingness to pay</li> <li>Run the campaign with 100 synthetic respondents</li> <li>Get detailed feedback: ratings, sentiment analysis, and verbatim responses</li> </ol> <p>The results include both quantitative metrics and rich qualitative insights—the kind of "why" explanations that make survey data actionable.</p> <h2 id="pricing">Pricing<a class="zola-anchor" href="#pricing" aria-label="Anchor link for: pricing" style="visibility: hidden;"></a> </h2> <p>PaperPin has a generous free tier:</p> <ul> <li><strong>Free:</strong> 10 respondents per survey, 5 surveys per month</li> <li><strong>Plus ($9/mo):</strong> 100 respondents, 50 surveys, premium AI models</li> <li><strong>Pro ($29/mo):</strong> 1,000 respondents, 500 surveys, export reports</li> <li><strong>Ultra ($99/mo):</strong> 100,000 respondents, 5,000 surveys, priority support</li> </ul> <p>No credit card required to start. You get 500 free credits to try it out.</p> <h2 id="try-it">Try It<a class="zola-anchor" href="#try-it" aria-label="Anchor link for: try-it" style="visibility: hidden;"></a> </h2> <p>If you're building products and need faster feedback loops, give <a rel="noopener nofollow noreferrer" target="_blank" href="https://paperpin.ai">PaperPin</a> a try. I'd love to hear what you think.</p> <p>The goal is simple: smart surveys, better decisions.</p> 2025-11-29T00:00:00+00:00 2025-11-29T00:00:00+00:00 https://rajnandan.com/posts/rogue-ai-agents-observability-trust/ <p>AI agents don't just chat. They reason, plan, call tools, and act. That's useful in customer support, supply chain, and IT operations. It's also risky. In production, agents can go "rogue" in ways that are hard to see until damage is done.</p> <p>Here's how it works. An agent makes a call you can't explain. It produces different outputs for the same input. Or it fails silently between steps. When that happens, debugging is guesswork, compliance is shaky, and trust erodes.</p> <p>Observability is how you fix that.</p> <h2 id="the-three-pillars">The Three Pillars<a class="zola-anchor" href="#the-three-pillars" aria-label="Anchor link for: the-three-pillars" style="visibility: hidden;"></a> </h2> <h3 id="1-decision-tracing">1. Decision Tracing<a class="zola-anchor" href="#1-decision-tracing" aria-label="Anchor link for: 1-decision-tracing" style="visibility: hidden;"></a> </h3> <p>Trace the path from input to output through every intermediate step. This includes prompts, retrieved context, tool calls, responses, and state changes. You're building a chain of evidence: what the agent saw, what it decided, and why.</p> <h3 id="2-behavioral-monitoring">2. Behavioral Monitoring<a class="zola-anchor" href="#2-behavioral-monitoring" aria-label="Anchor link for: 2-behavioral-monitoring" style="visibility: hidden;"></a> </h3> <p>Watch how the agent behaves, not just whether it runs. Look for loops, anomalies, and risky patterns:</p> <ul> <li>Infinite or long planning loops</li> <li>Repeated tool calls with no progress</li> <li>Outputs outside policy (e.g., PII disclosure, off-policy actions)</li> <li>Sudden drift in confidence or retrieval quality</li> </ul> <h3 id="3-outcome-alignment">3. Outcome Alignment<a class="zola-anchor" href="#3-outcome-alignment" aria-label="Anchor link for: 3-outcome-alignment" style="visibility: hidden;"></a> </h3> <p>Start with intent. Did the agent deliver the outcome you asked for, given the input and context? Measure the result against ground truth, policy, and business goals.</p> <p>If the intent was "reset password without exposing PII," you check exactly that - not a vague "agent said success."</p> <h2 id="what-you-capture">What You Capture<a class="zola-anchor" href="#what-you-capture" aria-label="Anchor link for: what-you-capture" style="visibility: hidden;"></a> </h2> <p>Good observability starts with the right data:</p> <p><strong>Inputs and context:</strong></p> <ul> <li>User request</li> <li>System instructions</li> <li>Retrieved documents</li> <li>Prior state</li> </ul> <p><strong>Decisions and reasoning:</strong></p> <ul> <li>Plans and thought steps</li> <li>Selected tools and parameters passed</li> <li>Results returned</li> <li>If you gate "reasoning" for privacy, still log a machine-readable trail of actions and justifications</li> </ul> <p><strong>Outcomes:</strong></p> <ul> <li>Final answer</li> <li>Side effects (tickets created, refunds issued)</li> <li>Validations (policy checks, human approvals, metrics)</li> </ul> <p>Store this as structured events. Each event has a timestamp, actor (agent/tool), action, inputs, outputs, and status. Stitch events into a timeline. That timeline is your replay: a transparent trail you can analyze, compare across runs, and improve.</p> <h2 id="observability-monitoring">Observability ≠ Monitoring<a class="zola-anchor" href="#observability-monitoring" aria-label="Anchor link for: observability-monitoring" style="visibility: hidden;"></a> </h2> <p>Monitoring tracks raw signals: CPU load, token count, error rate. Useful, but blind.</p> <p>Observability adds context: the full decision trail. You don't just know something failed - you see where, how, and what it did before failing.</p> <p>That's the difference between:</p> <ul> <li><em>"We saw 500s"</em></li> <li><em>"On step 4, the agent misread the policy doc and chose the refund tool incorrectly"</em></li> </ul> <h2 id="a-practical-example">A Practical Example<a class="zola-anchor" href="#a-practical-example" aria-label="Anchor link for: a-practical-example" style="visibility: hidden;"></a> </h2> <p><strong>Use case:</strong> A customer support agent that processes return requests.</p> <p><strong>Input:</strong> "I want to return order #123 for a defective charger."</p> <p><strong>Context:</strong> Policy docs, order data, past conversations.</p> <p><strong>Decision steps:</strong></p> <ol> <li><strong>Plan:</strong> Verify warranty, check return window, find nearest drop-off</li> <li><strong>Tool calls:</strong> Fetch order #123 → read warranty doc → query logistics API</li> <li><strong>Checks:</strong> Confirm user identity, detect PII exposure, enforce refund limits</li> </ol> <p><strong>Outcome:</strong> Issue return label, schedule pickup, confirm to user.</p> <p><strong>Observability in action:</strong></p> <ul> <li><strong>Decision tracing:</strong> Every step logged; you can replay the run</li> <li><strong>Behavioral monitoring:</strong> Flagged a loop when logistics API timed out thrice</li> <li><strong>Outcome alignment:</strong> Compared final action with policy; if policy prohibits returns after 30 days and the agent approved one on day 45, it triggers review</li> </ul> <p>This helps you move faster: incident response, policy audits, and continuous tuning. You stop guessing. You start improving.</p> <h2 id="operating-model">Operating Model<a class="zola-anchor" href="#operating-model" aria-label="Anchor link for: operating-model" style="visibility: hidden;"></a> </h2> <p><strong>Instrumentation:</strong> Add logging at the agent, tool, and framework layers. Use a consistent schema.</p> <p><strong>Policy guardrails:</strong> Codify checks as first-class steps. Log pass/fail with reasons.</p> <p><strong>Replay and diff:</strong> Compare timelines between "good" and "bad" runs. Highlight divergent decisions.</p> <p><strong>Metrics with context:</strong> Count loops, failed tool calls, off-policy actions. Tie them back to specific steps.</p> <p><strong>Feedback:</strong> Attach human review outcomes to the timeline. Use them to retrain or adjust prompts/tools.</p> <p><strong>Privacy and compliance:</strong> Redact sensitive fields; retain trace fidelity without violating policy.</p> <h2 id="common-failure-patterns">Common Failure Patterns<a class="zola-anchor" href="#common-failure-patterns" aria-label="Anchor link for: common-failure-patterns" style="visibility: hidden;"></a> </h2> <table><thead><tr><th>Pattern</th><th>Symptom</th><th>Fix</th></tr></thead><tbody> <tr><td><strong>Silent failures</strong></td><td>Agent stops mid-chain with no surface error</td><td>Log step-level status and timeouts</td></tr> <tr><td><strong>Ambiguous outputs</strong></td><td>Multiple conflicting answers for the same input</td><td>Deterministic policies and post-hoc validators</td></tr> <tr><td><strong>Tool thrashing</strong></td><td>Repeated calls without progress</td><td>Retry budgets, backoff, and loop detectors</td></tr> <tr><td><strong>Context drift</strong></td><td>Wrong or stale documents</td><td>Retrieval quality signals and provenance logging</td></tr> </tbody></table> <h2 id="what-trust-looks-like">What "Trust" Looks Like<a class="zola-anchor" href="#what-trust-looks-like" aria-label="Anchor link for: what-trust-looks-like" style="visibility: hidden;"></a> </h2> <p>Trust isn't a slogan. It's repeatable behavior under scrutiny.</p> <ul> <li>You can <strong>explain</strong> how the agent decided</li> <li>You can <strong>prove</strong> outcomes match intent and policy</li> <li>You can <strong>detect and correct</strong> anomalies quickly</li> <li>You can <strong>improve</strong> the agent based on evidence, not hunches</li> </ul> <h2 id="quick-start-checklist">Quick Start Checklist<a class="zola-anchor" href="#quick-start-checklist" aria-label="Anchor link for: quick-start-checklist" style="visibility: hidden;"></a> </h2> <ul> <li><input disabled="" type="checkbox"/> Define intent for each agent task: inputs, allowed tools, acceptable outcomes</li> <li><input disabled="" type="checkbox"/> Log every step with timestamps, inputs, outputs, and status</li> <li><input disabled="" type="checkbox"/> Add validators: policy checks, PII filters, safety rules</li> <li><input disabled="" type="checkbox"/> Build a timeline view and a replay tool</li> <li><input disabled="" type="checkbox"/> Track behavior metrics: loops, retries, off-policy decisions</li> <li><input disabled="" type="checkbox"/> Run postmortems with the trace when incidents occur</li> <li><input disabled="" type="checkbox"/> Feed learnings back: update prompts, tools, and policies</li> </ul> <h2 id="the-takeaway">The Takeaway<a class="zola-anchor" href="#the-takeaway" aria-label="Anchor link for: the-takeaway" style="visibility: hidden;"></a> </h2> <p>Observability for AI agents isn't dashboards and metrics. It's the full picture: inputs, decisions, and outcomes, stitched into a timeline you can trust, analyze, and improve.</p> <p>That's how you operate autonomous systems reliably at scale.</p> 2025-11-28T00:00:00+00:00 2025-11-28T00:00:00+00:00 https://rajnandan.com/posts/stripe-agentic-commerce-protocol/ <p>Stripe recently announced the Agentic Commerce Protocol (ACP), a new standard designed to enable AI agents to discover merchants, negotiate terms, and complete purchases on behalf of users. This is a significant step toward a future where your AI assistant doesn't just recommend products—it actually buys them for you.</p> <h2 id="the-problem-acp-solves">The Problem ACP Solves<a class="zola-anchor" href="#the-problem-acp-solves" aria-label="Anchor link for: the-problem-acp-solves" style="visibility: hidden;"></a> </h2> <p>Today's AI agents can do a lot. They can browse the web, summarize content, write code, and even interact with APIs. But when it comes to making purchases, they hit a wall. Here's why:</p> <ol> <li><strong>Trust</strong>: How does a merchant know the AI agent has permission to spend a user's money?</li> <li><strong>Discovery</strong>: How does an agent find merchants that can fulfill a specific request?</li> <li><strong>Negotiation</strong>: How do agents and merchants agree on terms, pricing, and fulfillment?</li> <li><strong>Compliance</strong>: How do you handle authentication, fraud prevention, and regulatory requirements?</li> </ol> <p>ACP addresses all of these by providing a standardized protocol that sits between AI agents, merchants, and payment infrastructure.</p> <h2 id="architecture-overview">Architecture Overview<a class="zola-anchor" href="#architecture-overview" aria-label="Anchor link for: architecture-overview" style="visibility: hidden;"></a> </h2> <p>ACP introduces three key participants in every transaction:</p> <pre class="z-code"><code><span class="z-text z-plain">┌─────────────┐ ┌─────────────┐ ┌─────────────┐ </span><span class="z-text z-plain">│ AI Agent │────▶│ ACP │────▶│ Merchant │ </span><span class="z-text z-plain">│ (Claude, │ │ Protocol │ │ (Seller) │ </span><span class="z-text z-plain">│ GPT, etc) │◀────│ Layer │◀────│ │ </span><span class="z-text z-plain">└─────────────┘ └─────────────┘ └─────────────┘ </span><span class="z-text z-plain"> │ │ │ </span><span class="z-text z-plain"> │ ▼ │ </span><span class="z-text z-plain"> │ ┌─────────────┐ │ </span><span class="z-text z-plain"> └──────────▶│ Stripe │◀───────────┘ </span><span class="z-text z-plain"> │ (Payment │ </span><span class="z-text z-plain"> │ Rails) │ </span><span class="z-text z-plain"> └─────────────┘ </span></code></pre> <h3 id="the-agent">The Agent<a class="zola-anchor" href="#the-agent" aria-label="Anchor link for: the-agent" style="visibility: hidden;"></a> </h3> <p>The AI agent acts on behalf of the user. It could be a general-purpose assistant like Claude or a specialized agent for travel, shopping, or business procurement. The agent needs to:</p> <ul> <li>Understand user intent</li> <li>Discover relevant merchants via ACP</li> <li>Present options and get user confirmation</li> <li>Execute the transaction</li> </ul> <h3 id="the-protocol-layer">The Protocol Layer<a class="zola-anchor" href="#the-protocol-layer" aria-label="Anchor link for: the-protocol-layer" style="visibility: hidden;"></a> </h3> <p>ACP defines a standard way for agents and merchants to communicate. This includes:</p> <ul> <li><strong>Discovery endpoints</strong>: Merchants register their capabilities, inventory, and terms</li> <li><strong>Negotiation schemas</strong>: Structured formats for quotes, offers, and counter-offers</li> <li><strong>Transaction lifecycle</strong>: States for pending, confirmed, fulfilled, and disputed transactions</li> <li><strong>Permission scopes</strong>: Granular control over what an agent can do</li> </ul> <h3 id="the-merchant">The Merchant<a class="zola-anchor" href="#the-merchant" aria-label="Anchor link for: the-merchant" style="visibility: hidden;"></a> </h3> <p>Merchants expose their catalog and business logic through ACP-compliant endpoints. They define:</p> <ul> <li>What products/services are available</li> <li>Pricing and availability rules</li> <li>Fulfillment options</li> <li>Refund and cancellation policies</li> </ul> <h2 id="how-a-transaction-works">How a Transaction Works<a class="zola-anchor" href="#how-a-transaction-works" aria-label="Anchor link for: how-a-transaction-works" style="visibility: hidden;"></a> </h2> <p>Let's walk through a real example: booking a restaurant reservation.</p> <h3 id="step-1-user-intent">Step 1: User Intent<a class="zola-anchor" href="#step-1-user-intent" aria-label="Anchor link for: step-1-user-intent" style="visibility: hidden;"></a> </h3> <pre class="z-code"><code><span class="z-text z-plain">User: &quot;Book me a table for 4 at an Italian restaurant </span><span class="z-text z-plain"> near downtown SF this Saturday at 7pm&quot; </span></code></pre> <h3 id="step-2-agent-discovery">Step 2: Agent Discovery<a class="zola-anchor" href="#step-2-agent-discovery" aria-label="Anchor link for: step-2-agent-discovery" style="visibility: hidden;"></a> </h3> <p>The agent queries ACP's discovery layer:</p> <pre data-lang="json" class="language-json z-code"><code class="language-json" data-lang="json"><span class="z-source z-json"><span class="z-meta z-mapping z-json"><span class="z-punctuation z-section z-mapping z-begin z-json">{</span> </span></span><span class="z-source z-json"><span class="z-meta z-mapping z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>intent<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>restaurant_reservation<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>constraints<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-json"><span class="z-punctuation z-section z-mapping z-begin z-json">{</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>cuisine<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>italian<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>location<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-json"><span class="z-punctuation z-section z-mapping z-begin z-json">{</span> </span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>near<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>downtown San Francisco<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>radius_km<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-constant z-numeric z-integer z-decimal z-json">2</span> </span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> <span class="z-punctuation z-section z-mapping z-end z-json">}</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>party_size<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-constant z-numeric z-integer z-decimal z-json">4</span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>datetime<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>2024-11-30T19:00:00-08:00<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> <span class="z-punctuation z-section z-mapping z-end z-json">}</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>user_id<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>usr_abc123<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>agent_id<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>agent_xyz789<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span> </span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-section z-mapping z-end z-json">}</span></span> </span></code></pre> <h3 id="step-3-merchant-responses">Step 3: Merchant Responses<a class="zola-anchor" href="#step-3-merchant-responses" aria-label="Anchor link for: step-3-merchant-responses" style="visibility: hidden;"></a> </h3> <p>Multiple merchants respond with availability:</p> <pre data-lang="json" class="language-json z-code"><code class="language-json" data-lang="json"><span class="z-source z-json"><span class="z-meta z-mapping z-json"><span class="z-punctuation z-section z-mapping z-begin z-json">{</span> </span></span><span class="z-source z-json"><span class="z-meta z-mapping z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>merchant_id<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>merch_italian_place<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>offer<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-json"><span class="z-punctuation z-section z-mapping z-begin z-json">{</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>offer_id<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>off_12345<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>description<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>Table for 4 at Trattoria Milano<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>datetime<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>2024-11-30T19:00:00-08:00<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>deposit_required<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-json"><span class="z-punctuation z-section z-mapping z-begin z-json">{</span> </span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>amount<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-constant z-numeric z-integer z-decimal z-json">2500</span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>currency<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>usd<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span> </span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> <span class="z-punctuation z-section z-mapping z-end z-json">}</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>cancellation_policy<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>free_until_24h_before<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>valid_until<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>2024-11-28T23:59:59-08:00<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> <span class="z-punctuation z-section z-mapping z-end z-json">}</span></span> </span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-section z-mapping z-end z-json">}</span></span> </span></code></pre> <h3 id="step-4-user-confirmation">Step 4: User Confirmation<a class="zola-anchor" href="#step-4-user-confirmation" aria-label="Anchor link for: step-4-user-confirmation" style="visibility: hidden;"></a> </h3> <p>The agent presents options to the user and gets confirmation. This is a critical step—ACP requires explicit user consent for transactions above certain thresholds or outside pre-authorized categories.</p> <h3 id="step-5-transaction-execution">Step 5: Transaction Execution<a class="zola-anchor" href="#step-5-transaction-execution" aria-label="Anchor link for: step-5-transaction-execution" style="visibility: hidden;"></a> </h3> <p>Once confirmed, the agent sends a transaction request:</p> <pre data-lang="json" class="language-json z-code"><code class="language-json" data-lang="json"><span class="z-source z-json"><span class="z-meta z-mapping z-json"><span class="z-punctuation z-section z-mapping z-begin z-json">{</span> </span></span><span class="z-source z-json"><span class="z-meta z-mapping z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>action<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>accept_offer<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>offer_id<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>off_12345<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>payment_method<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>pm_card_visa<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>user_authorization<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-json"><span class="z-punctuation z-section z-mapping z-begin z-json">{</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>token<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>auth_tok_xxx<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>scope<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-punctuation z-section z-sequence z-begin z-json">[</span><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>restaurant_reservation<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-section z-sequence z-end z-json">]</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>max_amount<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-constant z-numeric z-integer z-decimal z-json">5000</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> <span class="z-punctuation z-section z-mapping z-end z-json">}</span></span> </span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-section z-mapping z-end z-json">}</span></span> </span></code></pre> <h3 id="step-6-fulfillment">Step 6: Fulfillment<a class="zola-anchor" href="#step-6-fulfillment" aria-label="Anchor link for: step-6-fulfillment" style="visibility: hidden;"></a> </h3> <p>The merchant confirms the reservation and ACP tracks the fulfillment state. If something goes wrong, there's a standardized dispute resolution process.</p> <h2 id="the-technical-bits">The Technical Bits<a class="zola-anchor" href="#the-technical-bits" aria-label="Anchor link for: the-technical-bits" style="visibility: hidden;"></a> </h2> <h3 id="permission-model">Permission Model<a class="zola-anchor" href="#permission-model" aria-label="Anchor link for: permission-model" style="visibility: hidden;"></a> </h3> <p>ACP uses a hierarchical permission system. Users grant agents specific scopes:</p> <pre data-lang="json" class="language-json z-code"><code class="language-json" data-lang="json"><span class="z-source z-json"><span class="z-meta z-mapping z-json"><span class="z-punctuation z-section z-mapping z-begin z-json">{</span> </span></span><span class="z-source z-json"><span class="z-meta z-mapping z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>agent_permissions<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-json"><span class="z-punctuation z-section z-mapping z-begin z-json">{</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>agent_id<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>agent_xyz789<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>scopes<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-punctuation z-section z-sequence z-begin z-json">[</span> </span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"> <span class="z-meta z-mapping z-json"><span class="z-punctuation z-section z-mapping z-begin z-json">{</span> </span></span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-meta z-mapping z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>category<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>food_and_dining<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>max_transaction<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-constant z-numeric z-integer z-decimal z-json">10000</span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>max_daily<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-constant z-numeric z-integer z-decimal z-json">50000</span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>requires_confirmation<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-constant z-language z-json">false</span> </span></span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-meta z-mapping z-value z-json"> <span class="z-punctuation z-section z-mapping z-end z-json">}</span></span><span class="z-punctuation z-separator z-sequence z-json">,</span> </span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"> <span class="z-meta z-mapping z-json"><span class="z-punctuation z-section z-mapping z-begin z-json">{</span> </span></span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-meta z-mapping z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>category<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>travel<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>max_transaction<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-constant z-numeric z-integer z-decimal z-json">100000</span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>requires_confirmation<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-constant z-language z-json">true</span> </span></span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"><span class="z-meta z-mapping z-value z-json"> <span class="z-punctuation z-section z-mapping z-end z-json">}</span></span> </span></span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-sequence z-json"> <span class="z-punctuation z-section z-sequence z-end z-json">]</span></span><span class="z-punctuation z-separator z-mapping z-pair z-json">,</span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> </span><span class="z-meta z-mapping z-key z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>expires_at<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span></span><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-separator z-mapping z-key-value z-json">:</span> </span><span class="z-meta z-mapping z-value z-json"><span class="z-string z-quoted z-double z-json"><span class="z-punctuation z-definition z-string z-begin z-json">&quot;</span>2025-11-28T00:00:00Z<span class="z-punctuation z-definition z-string z-end z-json">&quot;</span></span> </span></span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-meta z-mapping z-value z-json"> <span class="z-punctuation z-section z-mapping z-end z-json">}</span></span> </span></span><span class="z-source z-json"><span class="z-meta z-mapping z-value z-json"><span class="z-punctuation z-section z-mapping z-end z-json">}</span></span> </span></code></pre> <p>This means you could let your agent book lunch without asking, but require confirmation for flight bookings.</p> <h3 id="mcp-integration">MCP Integration<a class="zola-anchor" href="#mcp-integration" aria-label="Anchor link for: mcp-integration" style="visibility: hidden;"></a> </h3> <p>ACP is designed to work with Anthropic's Model Context Protocol (MCP). In fact, Stripe has released an MCP server that exposes ACP functionality as tools that any MCP-compatible agent can use.</p> <pre data-lang="typescript" class="language-typescript z-code"><code class="language-typescript" data-lang="typescript"><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts"> Example MCP tool definition for ACP</span> </span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"> <span class="z-entity z-name z-label z-ts">name</span><span class="z-punctuation z-separator z-label z-ts">:</span> <span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>acp_discover_merchants<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"> <span class="z-entity z-name z-label z-ts">description</span><span class="z-punctuation z-separator z-label z-ts">:</span> <span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>Find merchants that can fulfill a purchase intent<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"> <span class="z-entity z-name z-label z-ts">parameters</span><span class="z-punctuation z-separator z-label z-ts">:</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-entity z-name z-label z-ts">intent</span><span class="z-punctuation z-separator z-label z-ts">:</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-entity z-name z-label z-ts">type</span><span class="z-punctuation z-separator z-label z-ts">:</span> <span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>string<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span> <span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-entity z-name z-label z-ts">constraints</span><span class="z-punctuation z-separator z-label z-ts">:</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-entity z-name z-label z-ts">type</span><span class="z-punctuation z-separator z-label z-ts">:</span> <span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>object<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span> <span class="z-punctuation z-definition z-block z-ts">}</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-entity z-name z-label z-ts">location</span><span class="z-punctuation z-separator z-label z-ts">:</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> <span class="z-entity z-name z-label z-ts">type</span><span class="z-punctuation z-separator z-label z-ts">:</span> <span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>object<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-entity z-name z-label z-ts">optional</span><span class="z-punctuation z-separator z-label z-ts">:</span> <span class="z-constant z-language z-boolean z-true z-ts">true</span> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span><span class="z-source z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></code></pre> <p>This means if you're building an agent using Claude, you can add ACP capabilities by connecting to Stripe's MCP server.</p> <h3 id="idempotency-and-state-management">Idempotency and State Management<a class="zola-anchor" href="#idempotency-and-state-management" aria-label="Anchor link for: idempotency-and-state-management" style="visibility: hidden;"></a> </h3> <p>E-commerce is messy. Networks fail, retries happen, and state gets out of sync. ACP handles this with:</p> <ul> <li><strong>Idempotency keys</strong>: Every transaction request includes a unique key. Retrying with the same key won't create duplicate charges.</li> <li><strong>State machine</strong>: Transactions follow a strict state machine with well-defined transitions</li> <li><strong>Webhooks</strong>: Merchants and agents receive real-time updates about state changes</li> </ul> <pre class="z-code"><code><span class="z-text z-plain">PENDING ──▶ CONFIRMED ──▶ FULFILLED </span><span class="z-text z-plain"> │ │ │ </span><span class="z-text z-plain"> ▼ ▼ ▼ </span><span class="z-text z-plain">CANCELLED REFUNDED DISPUTED </span></code></pre> <h2 id="why-this-matters">Why This Matters<a class="zola-anchor" href="#why-this-matters" aria-label="Anchor link for: why-this-matters" style="visibility: hidden;"></a> </h2> <h3 id="for-developers">For Developers<a class="zola-anchor" href="#for-developers" aria-label="Anchor link for: for-developers" style="visibility: hidden;"></a> </h3> <p>If you're building AI agents, ACP gives you a standardized way to add commerce capabilities without building payment infrastructure from scratch. You focus on the agent logic; Stripe handles the money movement.</p> <h3 id="for-merchants">For Merchants<a class="zola-anchor" href="#for-merchants" aria-label="Anchor link for: for-merchants" style="visibility: hidden;"></a> </h3> <p>You get access to a new distribution channel. As AI agents become more prevalent, being discoverable and transactable via ACP could be like being searchable on Google in the early 2000s.</p> <h3 id="for-users">For Users<a class="zola-anchor" href="#for-users" aria-label="Anchor link for: for-users" style="visibility: hidden;"></a> </h3> <p>The promise is convenience with control. Your agent can handle routine purchases automatically while still requiring your approval for significant decisions.</p> <h2 id="getting-started">Getting Started<a class="zola-anchor" href="#getting-started" aria-label="Anchor link for: getting-started" style="visibility: hidden;"></a> </h2> <p>Stripe's ACP is currently in beta. To start experimenting:</p> <ol> <li><strong>Sign up for the beta</strong> at stripe.com/acp</li> <li><strong>Install the MCP server</strong> if you're building MCP-compatible agents:<pre data-lang="bash" class="language-bash z-code"><code class="language-bash" data-lang="bash"><span class="z-source z-shell z-bash"><span class="z-meta z-function-call z-shell"><span class="z-variable z-function z-shell">npm</span></span><span class="z-meta z-function-call z-arguments z-shell"> install @stripe/acp-mcp-server</span> </span></code></pre> </li> <li><strong>Read the spec</strong>: The protocol specification is open and available for review</li> </ol> <h2 id="the-bigger-picture">The Bigger Picture<a class="zola-anchor" href="#the-bigger-picture" aria-label="Anchor link for: the-bigger-picture" style="visibility: hidden;"></a> </h2> <p>ACP is part of a broader trend toward AI agents that can take real-world actions. We're moving from AI that <em>tells</em> you what to do to AI that <em>does</em> things for you.</p> <p>But with great power comes great responsibility. The permission model, user confirmation requirements, and dispute resolution mechanisms in ACP reflect an understanding that autonomous spending needs guardrails.</p> <p>The interesting question isn't whether AI agents will make purchases for us—they will. The question is how we build the infrastructure to make this safe, trustworthy, and beneficial for everyone involved.</p> <p>Stripe's bet is that whoever builds the standard rails for agentic commerce will be well-positioned in this future. With ACP, they're making that bet concrete.</p> <hr /> <p><em>Disclaimer: This analysis is based on publicly available information about Stripe's ACP announcement. Implementation details may vary as the protocol evolves.</em></p> 2024-11-15T00:00:00+00:00 2024-11-15T00:00:00+00:00 https://rajnandan.com/posts/2x-engineer/ <p>We often hear about 10x engineers but today I want to talk about a 2x engineer. More specifically, I want to talk about what are those two qualities that make a 2x engineer.</p> <h2 id="quality-1-decision-making">Quality 1: Decision Making<a class="zola-anchor" href="#quality-1-decision-making" aria-label="Anchor link for: quality-1-decision-making" style="visibility: hidden;"></a> </h2> <p>Decision making is important in software development. You would often find yourself in a room with 10 other engineers and managers discussing about a new feature or a project that has to implemented. There would be a point in the meeting that a crucial path has to be selected out of let us say 2 different options. You would have to make a decision and it has to be a time bound exercise.</p> <p>This is where a 2x engineer would step up and point out the pros and cons of each path and make a decision. It is important to make sure that you are not just making a decision but also explaining why you made that decision.</p> <p>Your Decision should rely on:</p> <p><strong>Past experiences:</strong> The capability to make a decision comes with experience and knowledge. You would have to have faced similar problems previously even though not entirely similar but you should be able to draw an analogy. Depending on how the past decisions have turned out, you would be able to make a better decision this time.</p> <p><strong>Reversibility:</strong> The decision you make should be reversible. If you are not sure about the decision you are making, make sure that you can reverse it. Out of 100 things that you would have to decide on, there would just one or two that would not be reversible. At first you would find that a decision might be irreversible but if you think hard enough, if you are creative enough, you would find a way to reverse it.</p> <p><strong>Not everything has to be grandiose:</strong> There are people who job is to make sure that everything looks grandiose. But reality is that more often than not things are simpler, decisions are simpler. Your job is to identify those simpler decisions and make them and make them instantly without needing a meeting to discuss about it.</p> <h2 id="quality-2-communication">Quality 2: Communication<a class="zola-anchor" href="#quality-2-communication" aria-label="Anchor link for: quality-2-communication" style="visibility: hidden;"></a> </h2> <p>Communication is the key to success. You might have heard this a lot of times but it is true. You might be the best engineer in the world but if you cannot communicate your ideas, your thoughts, your decisions, your code, you are not going to be successful.</p> <p><strong>Writing:</strong> Writing is the most important form of communication. You would have to write a lot of emails, a lot of documentation, a lot of code. If you are not able to write clearly, you are not going to be successful. You would have to write in a way that the reader understands what you are trying to convey. You would have to write in a way that the reader does not have to ask you questions about what you wrote.</p> <p><strong>Speaking:</strong> Speaking is another form of communication. You would have to speak in meetings, you would have to speak in interviews, you would have to speak in presentations. If you are not able to speak clearly, you are not going to be successful. You would have to speak in a way that the listener understands what you are trying to convey. You would have to speak in a way that the listener does not have to ask you questions about what you spoke.</p> <p><strong>Listening:</strong> Listening is another form of communication. You would have to listen to your peers, your managers, your reports. If you are not able to listen clearly, you are not going to be successful. You would have to listen in a way that you understand what the other person is trying to convey. You would have to listen in a way that you do not have to ask questions about what the other person spoke.</p> <p><strong>Timelines:</strong> Give timelines to anything that is asked of you. It may not be favorable to the person asking but it is important to give a timeline.</p> 2024-07-18T00:00:00+00:00 2024-07-18T00:00:00+00:00 https://rajnandan.com/posts/promql-examples/ <p>PromQL is a powerful query language that allows you to query metrics from Prometheus. Here are some practical examples that you can use to query metrics from Prometheus.</p> <h2 id="working-with-counters">Working with counters<a class="zola-anchor" href="#working-with-counters" aria-label="Anchor link for: working-with-counters" style="visibility: hidden;"></a> </h2> <p>Let's assume we have a metric called <code>http_request_duration_seconds_count</code> that records the total number of HTTP requests made to a server. The metric has the following labels:</p> <ul> <li><code>method</code>: The HTTP method used for the request (e.g., GET, POST, PUT, DELETE).</li> <li><code>status</code>: The HTTP status code returned by the server (e.g., 200, 404, 500).</li> <li><code>handler</code>: The handler function that processed the request (e.g., <code>/api/v1/users</code>, <code>/api/v1/posts</code>).</li> </ul> <h3 id="time-durations">Time Durations<a class="zola-anchor" href="#time-durations" aria-label="Anchor link for: time-durations" style="visibility: hidden;"></a> </h3> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql"><span class="z-comment z-line z-number-sign z-sql"><span class="z-punctuation z-definition z-comment z-sql">#</span> Count the total number of HTTP requests in the last 1 second </span></span><span class="z-source z-sql"><span class="z-support z-function z-aggregate z-sql">sum</span>(rate(http_request_duration_seconds_count)) </span></code></pre> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql"><span class="z-comment z-line z-number-sign z-sql"><span class="z-punctuation z-definition z-comment z-sql">#</span> Count the total number of HTTP requests in the last 15 minutes </span></span><span class="z-source z-sql"><span class="z-support z-function z-aggregate z-sql">sum</span>(rate(http_request_duration_seconds_count[15m]) <span class="z-variable z-language z-star z-sql">*</span> <span class="z-constant z-numeric z-sql">60</span> <span class="z-variable z-language z-star z-sql">*</span> <span class="z-constant z-numeric z-sql">15</span>) </span></code></pre> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql"><span class="z-comment z-line z-number-sign z-sql"><span class="z-punctuation z-definition z-comment z-sql">#</span> Count the total number of HTTP requests in the last 1 day </span></span><span class="z-source z-sql"><span class="z-support z-function z-aggregate z-sql">sum</span>(rate(http_request_duration_seconds_count[1d]) <span class="z-variable z-language z-star z-sql">*</span> <span class="z-constant z-numeric z-sql">60</span> <span class="z-variable z-language z-star z-sql">*</span> <span class="z-constant z-numeric z-sql">60</span> <span class="z-variable z-language z-star z-sql">*</span> <span class="z-constant z-numeric z-sql">24</span>) </span></code></pre> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql"><span class="z-comment z-line z-number-sign z-sql"><span class="z-punctuation z-definition z-comment z-sql">#</span> Count the total number of HTTP requests in the last 1 week </span></span><span class="z-source z-sql"><span class="z-support z-function z-aggregate z-sql">sum</span>(rate(http_request_duration_seconds_count[1w]) <span class="z-variable z-language z-star z-sql">*</span> <span class="z-constant z-numeric z-sql">60</span> <span class="z-variable z-language z-star z-sql">*</span> <span class="z-constant z-numeric z-sql">60</span> <span class="z-variable z-language z-star z-sql">*</span> <span class="z-constant z-numeric z-sql">24</span> <span class="z-variable z-language z-star z-sql">*</span> <span class="z-constant z-numeric z-sql">7</span>) </span></code></pre> <p>The idea for these queries are simple - you are calculating the rate of change of the counter metric over a specific time period and then multiplying it by the number of seconds in that time period to get the total count.</p> <h3 id="group-by">Group by<a class="zola-anchor" href="#group-by" aria-label="Anchor link for: group-by" style="visibility: hidden;"></a> </h3> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql"><span class="z-comment z-line z-number-sign z-sql"><span class="z-punctuation z-definition z-comment z-sql">#</span> Count the total number of HTTP requests by method in the last 1 minute </span></span><span class="z-source z-sql"><span class="z-support z-function z-aggregate z-sql">sum</span>(rate(http_request_duration_seconds_count[1m] <span class="z-variable z-language z-star z-sql">*</span> <span class="z-constant z-numeric z-sql">60</span>)) by (method) </span></code></pre> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql"><span class="z-comment z-line z-number-sign z-sql"><span class="z-punctuation z-definition z-comment z-sql">#</span> Count the total number of HTTP requests by status code where handler is equal to /users in the last 5 minutes </span></span><span class="z-source z-sql"><span class="z-support z-function z-aggregate z-sql">sum</span>(rate(http_request_duration_seconds_count{handler <span class="z-keyword z-operator z-comparison z-sql">=</span> <span class="z-string z-quoted z-double z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&quot;</span>/users<span class="z-punctuation z-definition z-string z-end z-sql">&quot;</span></span>}[5m] <span class="z-variable z-language z-star z-sql">*</span> <span class="z-constant z-numeric z-sql">300</span>)) by (status) </span></code></pre> <h3 id="greater-than">Greater than<a class="zola-anchor" href="#greater-than" aria-label="Anchor link for: greater-than" style="visibility: hidden;"></a> </h3> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql"><span class="z-comment z-line z-number-sign z-sql"><span class="z-punctuation z-definition z-comment z-sql">#</span> Count the total number of HTTP requests by method in the last 1 minute if the count is greater than 50 </span></span><span class="z-source z-sql"><span class="z-support z-function z-aggregate z-sql">sum</span>(rate(http_request_duration_seconds_count[1m] <span class="z-variable z-language z-star z-sql">*</span> <span class="z-constant z-numeric z-sql">60</span>)) by (method) <span class="z-keyword z-operator z-comparison z-sql">&gt;</span> <span class="z-constant z-numeric z-sql">50</span> </span></code></pre> <h3 id="less-than">Less than<a class="zola-anchor" href="#less-than" aria-label="Anchor link for: less-than" style="visibility: hidden;"></a> </h3> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql"><span class="z-comment z-line z-number-sign z-sql"><span class="z-punctuation z-definition z-comment z-sql">#</span> Count the total number of HTTP requests by method in the last 1 minute if the count is less than 100 </span></span><span class="z-source z-sql"><span class="z-support z-function z-aggregate z-sql">sum</span>(rate(http_request_duration_seconds_count[1m] <span class="z-variable z-language z-star z-sql">*</span> <span class="z-constant z-numeric z-sql">60</span>)) by (method) <span class="z-keyword z-operator z-comparison z-sql">&lt;</span> <span class="z-constant z-numeric z-sql">100</span> </span></code></pre> <h3 id="within-range">Within Range<a class="zola-anchor" href="#within-range" aria-label="Anchor link for: within-range" style="visibility: hidden;"></a> </h3> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql"><span class="z-comment z-line z-number-sign z-sql"><span class="z-punctuation z-definition z-comment z-sql">#</span> Count the total number of HTTP requests by method in the last 1 minute if the count is between 50 and 100 </span></span><span class="z-source z-sql"><span class="z-support z-function z-aggregate z-sql">sum</span>(rate(http_request_duration_seconds_count[1m] <span class="z-variable z-language z-star z-sql">*</span> <span class="z-constant z-numeric z-sql">60</span>)) by (method) <span class="z-keyword z-operator z-comparison z-sql">&gt;</span> <span class="z-constant z-numeric z-sql">50</span> <span class="z-keyword z-operator z-comparison z-sql">&lt;</span> <span class="z-constant z-numeric z-sql">100</span> </span></code></pre> <h3 id="regular-expressions">Regular Expressions<a class="zola-anchor" href="#regular-expressions" aria-label="Anchor link for: regular-expressions" style="visibility: hidden;"></a> </h3> <p>Let us suppose we have rest apis for users like:</p> <ul> <li>POST /users</li> <li>GET /users/:user_id</li> <li>PUT /users/:user_id</li> <li>DELETE /users/:user_id</li> </ul> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql"><span class="z-comment z-line z-number-sign z-sql"><span class="z-punctuation z-definition z-comment z-sql">#</span> Count the total number of HTTP requests by handler and status code for similar handlers in the last 5 minutes </span></span><span class="z-source z-sql"><span class="z-support z-function z-aggregate z-sql">sum</span>(rate(http_request_duration_seconds_count{handler <span class="z-keyword z-operator z-comparison z-sql">=</span>~ <span class="z-string z-quoted z-double z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&quot;</span>/users.*<span class="z-punctuation z-definition z-string z-end z-sql">&quot;</span></span>}[5m] <span class="z-variable z-language z-star z-sql">*</span> <span class="z-constant z-numeric z-sql">300</span>)) by (status, handler) </span></code></pre> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql"><span class="z-comment z-line z-number-sign z-sql"><span class="z-punctuation z-definition z-comment z-sql">#</span> 5xx errors for the last 1 hour for each handler and method </span></span><span class="z-source z-sql"><span class="z-support z-function z-aggregate z-sql">sum</span>(rate(http_request_duration_seconds_count{status<span class="z-keyword z-operator z-comparison z-sql">=</span>~<span class="z-string z-quoted z-double z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&quot;</span>5..<span class="z-punctuation z-definition z-string z-end z-sql">&quot;</span></span>}[1h]) <span class="z-variable z-language z-star z-sql">*</span> <span class="z-constant z-numeric z-sql">60</span> <span class="z-variable z-language z-star z-sql">*</span> <span class="z-constant z-numeric z-sql">60</span>) by (handler, method) </span></code></pre> <h3 id="offset">Offset<a class="zola-anchor" href="#offset" aria-label="Anchor link for: offset" style="visibility: hidden;"></a> </h3> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql"><span class="z-comment z-line z-number-sign z-sql"><span class="z-punctuation z-definition z-comment z-sql">#</span> Count the total number of HTTP requests by handler and status code for similar handlers in the last to last 5 minutes </span></span><span class="z-source z-sql"><span class="z-support z-function z-aggregate z-sql">sum</span>(rate(http_request_duration_seconds_count{handler <span class="z-keyword z-operator z-comparison z-sql">=</span>~ <span class="z-string z-quoted z-double z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&quot;</span>/users.*<span class="z-punctuation z-definition z-string z-end z-sql">&quot;</span></span>}[5m] offset 5m) <span class="z-variable z-language z-star z-sql">*</span> <span class="z-constant z-numeric z-sql">300</span>) by (status, handler) </span></code></pre> <h3 id="delta">Delta<a class="zola-anchor" href="#delta" aria-label="Anchor link for: delta" style="visibility: hidden;"></a> </h3> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql"><span class="z-comment z-line z-number-sign z-sql"><span class="z-punctuation z-definition z-comment z-sql">#</span> Count the difference in the total number of HTTP requests by handler and status code for similar handlers in the last 5 minutes </span></span><span class="z-source z-sql"><span class="z-support z-function z-aggregate z-sql">sum</span>(rate(http_request_duration_seconds_count{handler <span class="z-keyword z-operator z-comparison z-sql">=</span>~ <span class="z-string z-quoted z-double z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&quot;</span>/users.*<span class="z-punctuation z-definition z-string z-end z-sql">&quot;</span></span>}[5m] <span class="z-variable z-language z-star z-sql">*</span> <span class="z-constant z-numeric z-sql">300</span>) <span class="z-keyword z-operator z-math z-sql">-</span> rate(http_request_duration_seconds_count{handler <span class="z-keyword z-operator z-comparison z-sql">=</span>~ <span class="z-string z-quoted z-double z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&quot;</span>/users.*<span class="z-punctuation z-definition z-string z-end z-sql">&quot;</span></span>}[5m] offset 5m) <span class="z-variable z-language z-star z-sql">*</span> <span class="z-constant z-numeric z-sql">300</span>) by (status, handler) </span></code></pre> <h3 id="percentage">Percentage<a class="zola-anchor" href="#percentage" aria-label="Anchor link for: percentage" style="visibility: hidden;"></a> </h3> <pre data-lang="sql" class="language-sql z-code"><code class="language-sql" data-lang="sql"><span class="z-source z-sql"><span class="z-comment z-line z-number-sign z-sql"><span class="z-punctuation z-definition z-comment z-sql">#</span> Calculate the percentage of 5xx errors for each handler in the last 1 hour </span></span><span class="z-source z-sql"><span class="z-support z-function z-aggregate z-sql">sum</span>(rate(http_request_duration_seconds_count{status<span class="z-keyword z-operator z-comparison z-sql">=</span>~<span class="z-string z-quoted z-double z-sql"><span class="z-punctuation z-definition z-string z-begin z-sql">&quot;</span>5..<span class="z-punctuation z-definition z-string z-end z-sql">&quot;</span></span>}[1h]) <span class="z-variable z-language z-star z-sql">*</span> <span class="z-constant z-numeric z-sql">60</span> <span class="z-variable z-language z-star z-sql">*</span> <span class="z-constant z-numeric z-sql">60</span>) <span class="z-keyword z-operator z-math z-sql">/</span> (<span class="z-support z-function z-aggregate z-sql">sum</span>(rate(http_request_duration_seconds_count[1h]) <span class="z-variable z-language z-star z-sql">*</span> <span class="z-constant z-numeric z-sql">60</span> <span class="z-variable z-language z-star z-sql">*</span> <span class="z-constant z-numeric z-sql">60</span>) by (handler) <span class="z-keyword z-operator z-comparison z-sql">&gt;</span> <span class="z-constant z-numeric z-sql">0</span>) <span class="z-variable z-language z-star z-sql">*</span> <span class="z-constant z-numeric z-sql">100</span> </span></code></pre> 2024-05-18T00:00:00+00:00 2024-05-18T00:00:00+00:00 https://rajnandan.com/posts/browser-window-messaging-ruto/ <p>At my work I have to build solutions around communication between browser and iframe or a popup window i.e. <code>&lt;iframe&gt;</code> or <code>window.open()</code>. I have to use <code>postMessage</code> JS API, and feel it is not very developer friendly. Consider the below situation:</p> <ol> <li>The parent window sends a message to the child window</li> <li>The child window receives this message</li> <li>The child calls a backend server with the message</li> <li>The child sends the response back to parent</li> <li>The parent wants to receive this backend response, does some calculation</li> <li>The parent then wants to send a second message to the child</li> <li>And the same thing repeats</li> </ol> <p>While this may seem trivial when you are doing simple HTTP calls from frontend but across windows something like this is much more complicated. We have also to keep in mind that security is our outmost concern.</p> <p>Let us try to code this thing. Parent is parent.com, child is at child.com.</p> <p>I am assuming parent has child as an iframe.</p> <h2 id="1-send-message-from-parent-to-child">1. Send Message from parent to child<a class="zola-anchor" href="#1-send-message-from-parent-to-child" aria-label="Anchor link for: 1-send-message-from-parent-to-child" style="visibility: hidden;"></a> </h2> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts">parent.com</span> </span><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">iframe</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-object z-property z-ts">contentWindow</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">postMessage</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">message</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>child.com<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></code></pre> <h2 id="2-receive-message-from-parent-in-child">2. Receive message from parent in child<a class="zola-anchor" href="#2-receive-message-from-parent-in-child" aria-label="Anchor link for: 2-receive-message-from-parent-in-child" style="visibility: hidden;"></a> </h2> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts">child.com</span> </span><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-support z-variable z-dom z-ts">window</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">addEventListener</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>message<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span><span class="z-meta z-arrow z-ts"> <span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">event</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">expectedOrigin</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>parent.com<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-support z-variable z-dom z-ts">event</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">origin</span> <span class="z-keyword z-operator z-comparison z-ts">!==</span> <span class="z-variable z-other z-readwrite z-ts">expectedOrigin</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">message</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-support z-variable z-dom z-ts">event</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-dom z-ts">data</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">doProcessingOfMessage</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">message</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></code></pre> <h2 id="3-child-does-some-processing">3. Child does some processing<a class="zola-anchor" href="#3-child-does-some-processing" aria-label="Anchor link for: 3-child-does-some-processing" style="visibility: hidden;"></a> </h2> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts">child.com</span> </span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-storage z-modifier z-async z-ts">async</span> <span class="z-storage z-type z-function z-ts">function</span> <span class="z-meta z-definition z-function z-ts"><span class="z-entity z-name z-function z-ts">doProcessingOfMessage</span></span><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">message</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">resp</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-keyword z-control z-flow z-ts">await</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">api</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">message</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> <span class="z-variable z-other z-readwrite z-ts">resp</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-function z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span> </span></code></pre> <h2 id="4-child-sends-response-back">4. Child sends response back<a class="zola-anchor" href="#4-child-sends-response-back" aria-label="Anchor link for: 4-child-sends-response-back" style="visibility: hidden;"></a> </h2> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts">child.com</span> </span><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-support z-variable z-dom z-ts">window</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">addEventListener</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>message<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span><span class="z-meta z-arrow z-ts"> <span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">event</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">expectedOrigin</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>parent.com<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-support z-variable z-dom z-ts">event</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">origin</span> <span class="z-keyword z-operator z-comparison z-ts">!==</span> <span class="z-variable z-other z-readwrite z-ts">expectedOrigin</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">message</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-support z-variable z-dom z-ts">event</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-dom z-ts">data</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">resp</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-keyword z-control z-flow z-ts">await</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">doProcessingOfMessage</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">message</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-support z-variable z-dom z-ts">window</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-dom z-ts">parent</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">postMessage</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">resp</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-string z-quoted z-single z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&#39;</span>parent.com<span class="z-punctuation z-definition z-string z-end z-ts">&#39;</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span><span class="z-meta z-brace z-round z-ts">)</span> </span></code></pre> <h2 id="5-parent-receives-the-message">5. Parent receives the message<a class="zola-anchor" href="#5-parent-receives-the-message" aria-label="Anchor link for: 5-parent-receives-the-message" style="visibility: hidden;"></a> </h2> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts">parent.com</span> </span><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-support z-variable z-dom z-ts">window</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">addEventListener</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>message<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span><span class="z-meta z-arrow z-ts"> <span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">event</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">expectedOrigin</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>child.com<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-support z-variable z-dom z-ts">event</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">origin</span> <span class="z-keyword z-operator z-comparison z-ts">!==</span> <span class="z-variable z-other z-readwrite z-ts">expectedOrigin</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">message</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-support z-variable z-dom z-ts">event</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-dom z-ts">data</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></code></pre> <h2 id="6-parent-sends-2nd-message">6. Parent sends 2nd message<a class="zola-anchor" href="#6-parent-sends-2nd-message" aria-label="Anchor link for: 6-parent-sends-2nd-message" style="visibility: hidden;"></a> </h2> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts">parent.com</span> </span><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-support z-variable z-dom z-ts">window</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">addEventListener</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>message<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span><span class="z-meta z-arrow z-ts"> <span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">event</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">expectedOrigin</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>child.com<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-conditional z-ts">if</span> <span class="z-meta z-brace z-round z-ts">(</span><span class="z-support z-variable z-dom z-ts">event</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">origin</span> <span class="z-keyword z-operator z-comparison z-ts">!==</span> <span class="z-variable z-other z-readwrite z-ts">expectedOrigin</span><span class="z-meta z-brace z-round z-ts">)</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-keyword z-control z-flow z-ts">return</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">message</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-support z-variable z-dom z-ts">event</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-variable z-property z-dom z-ts">data</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">newMessage</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">getNewMessage</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">message</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">iframe</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-object z-property z-ts">contentWindow</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">postMessage</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">message</span><span class="z-punctuation z-separator z-comma z-ts">,</span> <span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>child.com<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></code></pre> <p>As we can see the process is completely event based. And writing code to do something like this gets very messy.</p> <p>What if there was a nicer way to do this? Let us say:</p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts">parent.com</span> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">resp1</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-keyword z-control z-flow z-ts">await</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">sendMessageToChild</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">message1</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">message2</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-keyword z-control z-flow z-ts">await</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">someProcessing</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">resp1</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">resp2</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-keyword z-control z-flow z-ts">await</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">sendMessageToChild</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">message2</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></code></pre> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-comment z-line z-double-slash z-ts"><span class="z-punctuation z-definition z-comment z-ts">//</span></span><span class="z-comment z-line z-double-slash z-ts">child.com</span> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">message1</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-keyword z-control z-flow z-ts">await</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">receiveMessageFromParent</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"><span class="z-meta z-var z-expr z-ts"><span class="z-storage z-type z-ts">const</span> <span class="z-meta z-var-single-variable z-expr z-ts"><span class="z-meta z-definition z-variable z-ts"><span class="z-variable z-other z-constant z-ts">resp1</span></span> </span><span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-keyword z-control z-flow z-ts">await</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">someProcessing</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">message1</span><span class="z-meta z-brace z-round z-ts">)</span></span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span><span class="z-source z-ts"><span class="z-keyword z-control z-flow z-ts">await</span> <span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">sendMessageToParent</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-variable z-other z-readwrite z-ts">resp1</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></code></pre> <p>In the above scenario, the parent is the orchestrator and child is just responding to actions triggered by parent.</p> <p>In a different scenario a child can be the orchestrator and the parent would be just responding to actions triggered by the child.</p> <p>That means both the parent and child can send and receive. If parent is sender then child has to be receiver, and when parent is receiver the child has to be sender.</p> <h2 id="handle-retry">Handle Retry<a class="zola-anchor" href="#handle-retry" aria-label="Anchor link for: handle-retry" style="visibility: hidden;"></a> </h2> <p>We have to make sure that there are retries in communication. And also both parent and child should be ready to receive.</p> <h2 id="security">Security<a class="zola-anchor" href="#security" aria-label="Anchor link for: security" style="visibility: hidden;"></a> </h2> <p>We have to have origin checks in both child and domain. Ignore all messages if not received from where we want to receive.</p> <h2 id="solution">Solution<a class="zola-anchor" href="#solution" aria-label="Anchor link for: solution" style="visibility: hidden;"></a> </h2> <p>I think the best way to do this would be to build it like a client/server kind of a thing. Let us take an example of web client and Node.js Express server.</p> <p><strong>Client:</strong></p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-entity z-name z-function z-ts">fetch</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>/message<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span><span class="z-meta z-brace z-round z-ts">)</span> </span><span class="z-source z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-promise z-ts">then</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-arrow z-ts"><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">response</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> </span><span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">response</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-entity z-name z-function z-ts">json</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-meta z-brace z-round z-ts">)</span> </span><span class="z-source z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-promise z-ts">then</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-arrow z-ts"><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">data</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-support z-variable z-dom z-ts">document</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">getElementById</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>response<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">textContent</span> <span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-meta z-function-call z-ts"><span class="z-support z-constant z-json z-ts">JSON</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-json z-ts">stringify</span></span><span class="z-meta z-brace z-round z-ts">(</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-variable z-other z-readwrite z-ts">data</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-constant z-language z-null z-ts">null</span><span class="z-punctuation z-separator z-comma z-ts">,</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-constant z-numeric z-decimal z-ts">2</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span></span><span class="z-meta z-brace z-round z-ts">)</span> </span><span class="z-source z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-promise z-ts">catch</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-meta z-arrow z-ts"><span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">error</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-support z-variable z-dom z-ts">document</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">getElementById</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>response<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-variable z-other z-property z-ts">textContent</span> <span class="z-keyword z-operator z-assignment z-ts">=</span> <span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>Error: <span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span> <span class="z-keyword z-operator z-arithmetic z-ts">+</span> <span class="z-variable z-other z-readwrite z-ts">error</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-punctuation z-definition z-block z-ts">}</span></span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></code></pre> <p><strong>Server:</strong></p> <pre data-lang="js" class="language-js z-code"><code class="language-js" data-lang="js"><span class="z-source z-ts"><span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">app</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-dom z-ts">get</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>/message<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span><span class="z-punctuation z-separator z-comma z-ts">,</span><span class="z-meta z-arrow z-ts"> <span class="z-meta z-parameters z-ts"><span class="z-punctuation z-definition z-parameters z-begin z-ts">(</span><span class="z-variable z-parameter z-ts">req</span><span class="z-punctuation z-separator z-parameter z-ts">,</span> <span class="z-variable z-parameter z-ts">res</span><span class="z-punctuation z-definition z-parameters z-end z-ts">)</span></span> </span><span class="z-meta z-arrow z-ts"><span class="z-storage z-type z-function z-arrow z-ts">=&gt;</span> <span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">{</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"> <span class="z-meta z-function-call z-ts"><span class="z-variable z-other z-object z-ts">res</span><span class="z-punctuation z-accessor z-ts">.</span><span class="z-support z-function z-ts">send</span></span><span class="z-meta z-brace z-round z-ts">(</span><span class="z-string z-quoted z-double z-ts"><span class="z-punctuation z-definition z-string z-begin z-ts">&quot;</span>Hello, world!<span class="z-punctuation z-definition z-string z-end z-ts">&quot;</span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></span></span><span class="z-source z-ts"><span class="z-meta z-arrow z-ts"><span class="z-meta z-block z-ts"><span class="z-punctuation z-definition z-block z-ts">}</span></span></span><span class="z-meta z-brace z-round z-ts">)</span><span class="z-punctuation z-terminator z-statement z-ts">;</span> </span></code></pre> <p>I started to write the solution but soon realized it is not easy. So I have to take a more conservative approach.</p> <h2 id="introducing-ruto">Introducing Ruto<a class="zola-anchor" href="#introducing-ruto" aria-label="Anchor link for: introducing-ruto" style="visibility: hidden;"></a> </h2> <p>Ruto is a lightweight (4KB), fast and easy-to-use JS library that streamlines the communication between parent and child window (iframe/popup).</p> <p>It uses client-server design pattern to communicate between parent and child window. Any window can become the client or the server depending on who wants to send. It abstracts out the complications of postMessage API and provides a simple API to send and receive messages.</p> <p>You can visit the GitHub page <a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/rajnandan1/ruto">here</a>.</p> <h2 id="ruto-examples">Ruto Examples<a class="zola-anchor" href="#ruto-examples" aria-label="Anchor link for: ruto-examples" style="visibility: hidden;"></a> </h2> <h3 id="parent-to-iframe">Parent to iframe<a class="zola-anchor" href="#parent-to-iframe" aria-label="Anchor link for: parent-to-iframe" style="visibility: hidden;"></a> </h3> <p>Parent will send a message to iframe, Parent expects a response from the iframe within 10 secs.</p> <p><strong>Parent:</strong></p> <pre data-lang="html" class="language-html z-code"><code class="language-html" data-lang="html"><span class="z-text z-html z-basic"><span class="z-meta z-tag z-block z-any z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;</span><span class="z-entity z-name z-tag z-block z-any z-html">iframe</span> <span class="z-meta z-attribute-with-value z-id z-html"><span class="z-entity z-other z-attribute-name z-id z-html">id</span><span class="z-punctuation z-separator z-key-value z-html">=</span><span class="z-string z-quoted z-double z-html"><span class="z-punctuation z-definition z-string z-begin z-html">&quot;</span></span><span class="z-string z-quoted z-double z-html"><span class="z-meta z-toc-list z-id z-html">iframe</span><span class="z-punctuation z-definition z-string z-end z-html">&quot;</span></span></span> <span class="z-meta z-attribute-with-value z-html"><span class="z-entity z-other z-attribute-name z-html">src</span><span class="z-punctuation z-separator z-key-value z-html">=</span><span class="z-string z-quoted z-double z-html"><span class="z-punctuation z-definition z-string z-begin z-html">&quot;</span>https://example.com/child.html<span class="z-punctuation z-definition z-string z-end z-html">&quot;</span></span></span><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span><span class="z-meta z-tag z-block z-any z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;/</span><span class="z-entity z-name z-tag z-block z-any z-html">iframe</span><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span> </span><span class="z-text z-html z-basic"><span class="z-meta z-tag z-script z-begin z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;</span><span class="z-entity z-name z-tag z-script z-html">script</span></span><span class="z-meta z-tag z-script z-begin z-html"> <span class="z-meta z-attribute-with-value z-html"><span class="z-entity z-other z-attribute-name z-html">src</span><span class="z-punctuation z-separator z-key-value z-html">=</span><span class="z-string z-quoted z-double z-html"><span class="z-punctuation z-definition z-string z-begin z-html">&quot;</span>https://cdn.jsdelivr.net/gh/rajnandan1/ruto/dist/ruto.min.js<span class="z-punctuation z-definition z-string z-end z-html">&quot;</span></span></span><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span><span class="z-meta z-tag z-script z-end z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;/</span><span class="z-entity z-name z-tag z-script z-html">script</span><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span> </span><span class="z-text z-html z-basic"><span class="z-meta z-tag z-script z-begin z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;</span><span class="z-entity z-name z-tag z-script z-html">script</span></span><span class="z-meta z-tag z-script z-begin z-html"><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span> </span><span class="z-text z-html z-basic"> <span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-storage z-type z-js">const</span> <span class="z-meta z-binding z-name z-js"><span class="z-variable z-other z-readwrite z-js">iframe</span></span> <span class="z-keyword z-operator z-assignment z-js">=</span> <span class="z-meta z-function-call z-method z-js"><span class="z-support z-type z-object z-dom z-js">document</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-variable z-function z-js">getElementById</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-meta z-string z-js"><span class="z-string z-quoted z-double z-js"><span class="z-punctuation z-definition z-string z-begin z-js">&quot;</span>iframe<span class="z-punctuation z-definition z-string z-end z-js">&quot;</span></span></span><span class="z-punctuation z-section z-group z-end z-js">)</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> </span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"> <span class="z-storage z-type z-js">const</span> <span class="z-meta z-binding z-name z-js"><span class="z-variable z-other z-readwrite z-js">message</span></span> <span class="z-keyword z-operator z-assignment z-js">=</span> <span class="z-meta z-string z-js"><span class="z-string z-quoted z-double z-js"><span class="z-punctuation z-definition z-string z-begin z-js">&quot;</span>Hello from parent<span class="z-punctuation z-definition z-string z-end z-js">&quot;</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> </span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"> <span class="z-storage z-type z-js">const</span> <span class="z-meta z-binding z-name z-js"><span class="z-variable z-other z-readwrite z-js">options</span></span> <span class="z-keyword z-operator z-assignment z-js">=</span> <span class="z-meta z-mapping z-js"><span class="z-punctuation z-section z-block z-begin z-js">{</span> </span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-mapping z-js"> <span class="z-meta z-mapping z-key z-js">timeout</span><span class="z-punctuation z-separator z-key-value z-js">:</span> <span class="z-constant z-numeric z-integer z-decimal z-js">10000</span><span class="z-punctuation z-separator z-comma z-js">,</span> </span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-mapping z-js"> <span class="z-punctuation z-section z-block z-end z-js">}</span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> </span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"> </span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"> <span class="z-meta z-function-call z-method z-js"><span class="z-variable z-other z-readwrite z-js">ruto</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-variable z-function z-js">send</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-meta z-string z-js"><span class="z-string z-quoted z-double z-js"><span class="z-punctuation z-definition z-string z-begin z-js">&quot;</span>https://example.com/parent-to-iframe/topic1<span class="z-punctuation z-definition z-string z-end z-js">&quot;</span></span></span><span class="z-punctuation z-separator z-comma z-js">,</span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-variable z-other z-readwrite z-js">iframe</span><span class="z-punctuation z-separator z-comma z-js">,</span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-variable z-other z-readwrite z-js">message</span><span class="z-punctuation z-separator z-comma z-js">,</span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-variable z-other z-readwrite z-js">options</span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-punctuation z-section z-group z-end z-js">)</span></span> </span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"> </span><span class="z-punctuation z-accessor z-js">.</span><span class="z-meta z-function-call z-method z-js"><span class="z-variable z-function z-js">then</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-meta z-function z-js"></span><span class="z-meta z-function z-declaration z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-meta z-binding z-name z-js"><span class="z-variable z-parameter z-function z-js">response</span></span><span class="z-punctuation z-section z-group z-end z-js">)</span> <span class="z-storage z-type z-function z-arrow z-js">=&gt;</span></span><span class="z-meta z-function z-js"> <span class="z-meta z-block z-js"><span class="z-punctuation z-section z-block z-begin z-js">{</span> </span></span></span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"><span class="z-meta z-function z-js"><span class="z-meta z-block z-js"> <span class="z-meta z-function-call z-method z-js"><span class="z-support z-type z-object z-console z-js">console</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-support z-function z-console z-js">log</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-variable z-other z-readwrite z-js">response</span><span class="z-punctuation z-section z-group z-end z-js">)</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> <span class="z-comment z-line z-double-slash z-js"><span class="z-punctuation z-definition z-comment z-js">//</span>Hello from child </span></span></span></span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"><span class="z-meta z-function z-js"><span class="z-meta z-block z-js"> <span class="z-punctuation z-section z-block z-end z-js">}</span></span></span><span class="z-punctuation z-section z-group z-end z-js">)</span></span> </span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"> </span><span class="z-punctuation z-accessor z-js">.</span><span class="z-meta z-function-call z-method z-js"><span class="z-variable z-function z-js">catch</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-meta z-function z-js"></span><span class="z-meta z-function z-declaration z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-meta z-binding z-name z-js"><span class="z-variable z-parameter z-function z-js">error</span></span><span class="z-punctuation z-section z-group z-end z-js">)</span> <span class="z-storage z-type z-function z-arrow z-js">=&gt;</span></span><span class="z-meta z-function z-js"> <span class="z-meta z-block z-js"><span class="z-punctuation z-section z-block z-begin z-js">{</span> </span></span></span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"><span class="z-meta z-function z-js"><span class="z-meta z-block z-js"> <span class="z-meta z-function-call z-method z-js"><span class="z-support z-type z-object z-console z-js">console</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-support z-function z-console z-js">log</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-variable z-other z-readwrite z-js">error</span><span class="z-punctuation z-section z-group z-end z-js">)</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> </span></span></span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"><span class="z-meta z-function z-js"><span class="z-meta z-block z-js"> <span class="z-punctuation z-section z-block z-end z-js">}</span></span></span><span class="z-punctuation z-section z-group z-end z-js">)</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> </span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"></span></span><span class="z-meta z-tag z-script z-end z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;/</span><span class="z-entity z-name z-tag z-script z-html">script</span><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span> </span></code></pre> <p><strong>Child:</strong></p> <pre data-lang="html" class="language-html z-code"><code class="language-html" data-lang="html"><span class="z-text z-html z-basic"><span class="z-meta z-tag z-script z-begin z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;</span><span class="z-entity z-name z-tag z-script z-html">script</span></span><span class="z-meta z-tag z-script z-begin z-html"> <span class="z-meta z-attribute-with-value z-html"><span class="z-entity z-other z-attribute-name z-html">src</span><span class="z-punctuation z-separator z-key-value z-html">=</span><span class="z-string z-quoted z-double z-html"><span class="z-punctuation z-definition z-string z-begin z-html">&quot;</span>https://cdn.jsdelivr.net/gh/rajnandan1/ruto/dist/ruto.min.js<span class="z-punctuation z-definition z-string z-end z-html">&quot;</span></span></span><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span><span class="z-meta z-tag z-script z-end z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;/</span><span class="z-entity z-name z-tag z-script z-html">script</span><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span> </span><span class="z-text z-html z-basic"><span class="z-meta z-tag z-script z-begin z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;</span><span class="z-entity z-name z-tag z-script z-html">script</span></span><span class="z-meta z-tag z-script z-begin z-html"><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span> </span><span class="z-text z-html z-basic"> <span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-variable z-other z-readwrite z-js">ruto</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-variable z-function z-js">receive</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-meta z-string z-js"><span class="z-string z-quoted z-double z-js"><span class="z-punctuation z-definition z-string z-begin z-js">&quot;</span>/parent-to-iframe/topic1<span class="z-punctuation z-definition z-string z-end z-js">&quot;</span></span></span><span class="z-punctuation z-separator z-comma z-js">,</span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-support z-type z-object z-dom z-js">window</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-meta z-property z-object z-js">parent</span><span class="z-punctuation z-separator z-comma z-js">,</span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-meta z-function z-js"></span><span class="z-meta z-function z-declaration z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-meta z-binding z-name z-js"><span class="z-variable z-parameter z-function z-js">response</span></span><span class="z-punctuation z-separator z-parameter z-function z-js">,</span> <span class="z-meta z-binding z-name z-js"><span class="z-variable z-parameter z-function z-js">message</span></span><span class="z-punctuation z-section z-group z-end z-js">)</span> <span class="z-storage z-type z-function z-arrow z-js">=&gt;</span></span><span class="z-meta z-function z-js"> <span class="z-meta z-block z-js"><span class="z-punctuation z-section z-block z-begin z-js">{</span> </span></span></span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"><span class="z-meta z-function z-js"><span class="z-meta z-block z-js"> <span class="z-meta z-function-call z-method z-js"><span class="z-support z-type z-object z-console z-js">console</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-support z-function z-console z-js">log</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-variable z-other z-readwrite z-js">message</span><span class="z-punctuation z-section z-group z-end z-js">)</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> <span class="z-comment z-line z-double-slash z-js"><span class="z-punctuation z-definition z-comment z-js">//</span>Hello from parent </span></span></span></span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"><span class="z-meta z-function z-js"><span class="z-meta z-block z-js"> <span class="z-keyword z-control z-flow z-return z-js">return</span> <span class="z-meta z-function-call z-method z-js"><span class="z-variable z-other z-readwrite z-js">response</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-variable z-function z-js">send</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-meta z-string z-js"><span class="z-string z-quoted z-double z-js"><span class="z-punctuation z-definition z-string z-begin z-js">&quot;</span>Hello from child<span class="z-punctuation z-definition z-string z-end z-js">&quot;</span></span></span><span class="z-punctuation z-section z-group z-end z-js">)</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> </span></span></span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"><span class="z-meta z-function z-js"><span class="z-meta z-block z-js"> <span class="z-punctuation z-section z-block z-end z-js">}</span></span></span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-punctuation z-section z-group z-end z-js">)</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> </span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"></span></span><span class="z-meta z-tag z-script z-end z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;/</span><span class="z-entity z-name z-tag z-script z-html">script</span><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span> </span></code></pre> <h3 id="iframe-to-parent">Iframe to Parent<a class="zola-anchor" href="#iframe-to-parent" aria-label="Anchor link for: iframe-to-parent" style="visibility: hidden;"></a> </h3> <p>Iframe will send a message to parent, and expects a response from parent within 5 secs.</p> <p><strong>Parent:</strong></p> <pre data-lang="html" class="language-html z-code"><code class="language-html" data-lang="html"><span class="z-text z-html z-basic"><span class="z-meta z-tag z-block z-any z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;</span><span class="z-entity z-name z-tag z-block z-any z-html">iframe</span> <span class="z-meta z-attribute-with-value z-id z-html"><span class="z-entity z-other z-attribute-name z-id z-html">id</span><span class="z-punctuation z-separator z-key-value z-html">=</span><span class="z-string z-quoted z-double z-html"><span class="z-punctuation z-definition z-string z-begin z-html">&quot;</span></span><span class="z-string z-quoted z-double z-html"><span class="z-meta z-toc-list z-id z-html">iframe</span><span class="z-punctuation z-definition z-string z-end z-html">&quot;</span></span></span> <span class="z-meta z-attribute-with-value z-html"><span class="z-entity z-other z-attribute-name z-html">src</span><span class="z-punctuation z-separator z-key-value z-html">=</span><span class="z-string z-quoted z-double z-html"><span class="z-punctuation z-definition z-string z-begin z-html">&quot;</span>https://example.com/child.html<span class="z-punctuation z-definition z-string z-end z-html">&quot;</span></span></span><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span><span class="z-meta z-tag z-block z-any z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;/</span><span class="z-entity z-name z-tag z-block z-any z-html">iframe</span><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span> </span><span class="z-text z-html z-basic"><span class="z-meta z-tag z-script z-begin z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;</span><span class="z-entity z-name z-tag z-script z-html">script</span></span><span class="z-meta z-tag z-script z-begin z-html"> <span class="z-meta z-attribute-with-value z-html"><span class="z-entity z-other z-attribute-name z-html">src</span><span class="z-punctuation z-separator z-key-value z-html">=</span><span class="z-string z-quoted z-double z-html"><span class="z-punctuation z-definition z-string z-begin z-html">&quot;</span>https://cdn.jsdelivr.net/gh/rajnandan1/ruto/dist/ruto.min.js<span class="z-punctuation z-definition z-string z-end z-html">&quot;</span></span></span><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span><span class="z-meta z-tag z-script z-end z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;/</span><span class="z-entity z-name z-tag z-script z-html">script</span><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span> </span><span class="z-text z-html z-basic"><span class="z-meta z-tag z-script z-begin z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;</span><span class="z-entity z-name z-tag z-script z-html">script</span></span><span class="z-meta z-tag z-script z-begin z-html"><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span> </span><span class="z-text z-html z-basic"> <span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-storage z-type z-js">const</span> <span class="z-meta z-binding z-name z-js"><span class="z-variable z-other z-readwrite z-js">iframe</span></span> <span class="z-keyword z-operator z-assignment z-js">=</span> <span class="z-meta z-function-call z-method z-js"><span class="z-support z-type z-object z-dom z-js">document</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-variable z-function z-js">getElementById</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-meta z-string z-js"><span class="z-string z-quoted z-double z-js"><span class="z-punctuation z-definition z-string z-begin z-js">&quot;</span>iframe<span class="z-punctuation z-definition z-string z-end z-js">&quot;</span></span></span><span class="z-punctuation z-section z-group z-end z-js">)</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> </span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"> <span class="z-meta z-function-call z-method z-js"><span class="z-variable z-other z-readwrite z-js">ruto</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-variable z-function z-js">receive</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-meta z-string z-js"><span class="z-string z-quoted z-double z-js"><span class="z-punctuation z-definition z-string z-begin z-js">&quot;</span>/iframe-to-parent/topic1<span class="z-punctuation z-definition z-string z-end z-js">&quot;</span></span></span><span class="z-punctuation z-separator z-comma z-js">,</span> <span class="z-variable z-other z-readwrite z-js">iframe</span><span class="z-punctuation z-separator z-comma z-js">,</span> <span class="z-meta z-function z-js"></span><span class="z-meta z-function z-declaration z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-meta z-binding z-name z-js"><span class="z-variable z-parameter z-function z-js">response</span></span><span class="z-punctuation z-separator z-parameter z-function z-js">,</span> <span class="z-meta z-binding z-name z-js"><span class="z-variable z-parameter z-function z-js">message</span></span><span class="z-punctuation z-section z-group z-end z-js">)</span> <span class="z-storage z-type z-function z-arrow z-js">=&gt;</span></span><span class="z-meta z-function z-js"> <span class="z-meta z-block z-js"><span class="z-punctuation z-section z-block z-begin z-js">{</span> </span></span></span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"><span class="z-meta z-function z-js"><span class="z-meta z-block z-js"> <span class="z-meta z-function-call z-method z-js"><span class="z-support z-type z-object z-console z-js">console</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-support z-function z-console z-js">log</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-variable z-other z-readwrite z-js">message</span><span class="z-punctuation z-section z-group z-end z-js">)</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> <span class="z-comment z-line z-double-slash z-js"><span class="z-punctuation z-definition z-comment z-js">//</span>Hello from iframe </span></span></span></span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"><span class="z-meta z-function z-js"><span class="z-meta z-block z-js"> <span class="z-keyword z-control z-flow z-return z-js">return</span> <span class="z-meta z-function-call z-method z-js"><span class="z-variable z-other z-readwrite z-js">response</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-variable z-function z-js">send</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-meta z-string z-js"><span class="z-string z-quoted z-double z-js"><span class="z-punctuation z-definition z-string z-begin z-js">&quot;</span>Hello from parent<span class="z-punctuation z-definition z-string z-end z-js">&quot;</span></span></span><span class="z-punctuation z-section z-group z-end z-js">)</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> </span></span></span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"><span class="z-meta z-function z-js"><span class="z-meta z-block z-js"> <span class="z-punctuation z-section z-block z-end z-js">}</span></span></span><span class="z-punctuation z-section z-group z-end z-js">)</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> </span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"></span></span><span class="z-meta z-tag z-script z-end z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;/</span><span class="z-entity z-name z-tag z-script z-html">script</span><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span> </span></code></pre> <p><strong>Child:</strong></p> <pre data-lang="html" class="language-html z-code"><code class="language-html" data-lang="html"><span class="z-text z-html z-basic"><span class="z-meta z-tag z-script z-begin z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;</span><span class="z-entity z-name z-tag z-script z-html">script</span></span><span class="z-meta z-tag z-script z-begin z-html"> <span class="z-meta z-attribute-with-value z-html"><span class="z-entity z-other z-attribute-name z-html">src</span><span class="z-punctuation z-separator z-key-value z-html">=</span><span class="z-string z-quoted z-double z-html"><span class="z-punctuation z-definition z-string z-begin z-html">&quot;</span>https://cdn.jsdelivr.net/gh/rajnandan1/ruto/dist/ruto.min.js<span class="z-punctuation z-definition z-string z-end z-html">&quot;</span></span></span><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span><span class="z-meta z-tag z-script z-end z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;/</span><span class="z-entity z-name z-tag z-script z-html">script</span><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span> </span><span class="z-text z-html z-basic"><span class="z-meta z-tag z-script z-begin z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;</span><span class="z-entity z-name z-tag z-script z-html">script</span></span><span class="z-meta z-tag z-script z-begin z-html"><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span> </span><span class="z-text z-html z-basic"> <span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-storage z-type z-js">const</span> <span class="z-meta z-binding z-name z-js"><span class="z-variable z-other z-readwrite z-js">message</span></span> <span class="z-keyword z-operator z-assignment z-js">=</span> <span class="z-meta z-string z-js"><span class="z-string z-quoted z-double z-js"><span class="z-punctuation z-definition z-string z-begin z-js">&quot;</span>Hello from iframe<span class="z-punctuation z-definition z-string z-end z-js">&quot;</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> </span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"> <span class="z-storage z-type z-js">const</span> <span class="z-meta z-binding z-name z-js"><span class="z-variable z-other z-readwrite z-js">options</span></span> <span class="z-keyword z-operator z-assignment z-js">=</span> <span class="z-meta z-mapping z-js"><span class="z-punctuation z-section z-block z-begin z-js">{</span> </span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-mapping z-js"> <span class="z-meta z-mapping z-key z-js">timeout</span><span class="z-punctuation z-separator z-key-value z-js">:</span> <span class="z-constant z-numeric z-integer z-decimal z-js">5000</span><span class="z-punctuation z-separator z-comma z-js">,</span> </span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-mapping z-js"> <span class="z-punctuation z-section z-block z-end z-js">}</span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> </span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"> <span class="z-meta z-function-call z-method z-js"><span class="z-variable z-other z-readwrite z-js">ruto</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-variable z-function z-js">send</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-meta z-string z-js"><span class="z-string z-quoted z-double z-js"><span class="z-punctuation z-definition z-string z-begin z-js">&quot;</span>https://example.com/iframe-to-parent/topic1<span class="z-punctuation z-definition z-string z-end z-js">&quot;</span></span></span><span class="z-punctuation z-separator z-comma z-js">,</span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-support z-type z-object z-dom z-js">window</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-meta z-property z-object z-js">parent</span><span class="z-punctuation z-separator z-comma z-js">,</span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-variable z-other z-readwrite z-js">message</span><span class="z-punctuation z-separator z-comma z-js">,</span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-variable z-other z-readwrite z-js">options</span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-punctuation z-section z-group z-end z-js">)</span></span> </span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"> </span><span class="z-punctuation z-accessor z-js">.</span><span class="z-meta z-function-call z-method z-js"><span class="z-variable z-function z-js">then</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-meta z-function z-js"></span><span class="z-meta z-function z-declaration z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-meta z-binding z-name z-js"><span class="z-variable z-parameter z-function z-js">response</span></span><span class="z-punctuation z-section z-group z-end z-js">)</span> <span class="z-storage z-type z-function z-arrow z-js">=&gt;</span></span><span class="z-meta z-function z-js"> <span class="z-meta z-block z-js"><span class="z-punctuation z-section z-block z-begin z-js">{</span> </span></span></span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"><span class="z-meta z-function z-js"><span class="z-meta z-block z-js"> <span class="z-meta z-function-call z-method z-js"><span class="z-support z-type z-object z-console z-js">console</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-support z-function z-console z-js">log</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-variable z-other z-readwrite z-js">response</span><span class="z-punctuation z-section z-group z-end z-js">)</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> <span class="z-comment z-line z-double-slash z-js"><span class="z-punctuation z-definition z-comment z-js">//</span>Hello from parent </span></span></span></span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"><span class="z-meta z-function z-js"><span class="z-meta z-block z-js"> <span class="z-punctuation z-section z-block z-end z-js">}</span></span></span><span class="z-punctuation z-section z-group z-end z-js">)</span></span> </span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"> </span><span class="z-punctuation z-accessor z-js">.</span><span class="z-meta z-function-call z-method z-js"><span class="z-variable z-function z-js">catch</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-meta z-function z-js"></span><span class="z-meta z-function z-declaration z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-meta z-binding z-name z-js"><span class="z-variable z-parameter z-function z-js">error</span></span><span class="z-punctuation z-section z-group z-end z-js">)</span> <span class="z-storage z-type z-function z-arrow z-js">=&gt;</span></span><span class="z-meta z-function z-js"> <span class="z-meta z-block z-js"><span class="z-punctuation z-section z-block z-begin z-js">{</span> </span></span></span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"><span class="z-meta z-function z-js"><span class="z-meta z-block z-js"> <span class="z-meta z-function-call z-method z-js"><span class="z-support z-type z-object z-console z-js">console</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-support z-function z-console z-js">log</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-variable z-other z-readwrite z-js">error</span><span class="z-punctuation z-section z-group z-end z-js">)</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> </span></span></span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"><span class="z-meta z-function z-js"><span class="z-meta z-block z-js"> <span class="z-punctuation z-section z-block z-end z-js">}</span></span></span><span class="z-punctuation z-section z-group z-end z-js">)</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> </span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"></span></span><span class="z-meta z-tag z-script z-end z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;/</span><span class="z-entity z-name z-tag z-script z-html">script</span><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span> </span></code></pre> <h3 id="parent-to-popup">Parent to Popup<a class="zola-anchor" href="#parent-to-popup" aria-label="Anchor link for: parent-to-popup" style="visibility: hidden;"></a> </h3> <p>Parent will send a message to popup, and expects a response from popup within 5 secs.</p> <p><strong>Parent:</strong></p> <pre data-lang="html" class="language-html z-code"><code class="language-html" data-lang="html"><span class="z-text z-html z-basic"><span class="z-meta z-tag z-script z-begin z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;</span><span class="z-entity z-name z-tag z-script z-html">script</span></span><span class="z-meta z-tag z-script z-begin z-html"> <span class="z-meta z-attribute-with-value z-html"><span class="z-entity z-other z-attribute-name z-html">src</span><span class="z-punctuation z-separator z-key-value z-html">=</span><span class="z-string z-quoted z-double z-html"><span class="z-punctuation z-definition z-string z-begin z-html">&quot;</span>https://cdn.jsdelivr.net/gh/rajnandan1/ruto/dist/ruto.min.js<span class="z-punctuation z-definition z-string z-end z-html">&quot;</span></span></span><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span><span class="z-meta z-tag z-script z-end z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;/</span><span class="z-entity z-name z-tag z-script z-html">script</span><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span> </span><span class="z-text z-html z-basic"><span class="z-meta z-tag z-script z-begin z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;</span><span class="z-entity z-name z-tag z-script z-html">script</span></span><span class="z-meta z-tag z-script z-begin z-html"><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span> </span><span class="z-text z-html z-basic"> <span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-storage z-type z-js">const</span> <span class="z-meta z-binding z-name z-js"><span class="z-variable z-other z-readwrite z-js">popup</span></span> <span class="z-keyword z-operator z-assignment z-js">=</span> <span class="z-meta z-function-call z-method z-js"><span class="z-support z-type z-object z-dom z-js">window</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-variable z-function z-js">open</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-meta z-string z-js"><span class="z-string z-quoted z-double z-js"><span class="z-punctuation z-definition z-string z-begin z-js">&quot;</span>https://example.com/popup.html<span class="z-punctuation z-definition z-string z-end z-js">&quot;</span></span></span><span class="z-punctuation z-separator z-comma z-js">,</span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-meta z-string z-js"><span class="z-string z-quoted z-double z-js"><span class="z-punctuation z-definition z-string z-begin z-js">&quot;</span>popup<span class="z-punctuation z-definition z-string z-end z-js">&quot;</span></span></span><span class="z-punctuation z-separator z-comma z-js">,</span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-meta z-string z-js"><span class="z-string z-quoted z-double z-js"><span class="z-punctuation z-definition z-string z-begin z-js">&quot;</span>width=600,height=400<span class="z-punctuation z-definition z-string z-end z-js">&quot;</span></span></span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-punctuation z-section z-group z-end z-js">)</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> </span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"> <span class="z-storage z-type z-js">const</span> <span class="z-meta z-binding z-name z-js"><span class="z-variable z-other z-readwrite z-js">message</span></span> <span class="z-keyword z-operator z-assignment z-js">=</span> <span class="z-meta z-string z-js"><span class="z-string z-quoted z-double z-js"><span class="z-punctuation z-definition z-string z-begin z-js">&quot;</span>Hello from parent<span class="z-punctuation z-definition z-string z-end z-js">&quot;</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> </span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"> <span class="z-storage z-type z-js">const</span> <span class="z-meta z-binding z-name z-js"><span class="z-variable z-other z-readwrite z-js">options</span></span> <span class="z-keyword z-operator z-assignment z-js">=</span> <span class="z-meta z-mapping z-js"><span class="z-punctuation z-section z-block z-begin z-js">{</span> </span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-mapping z-js"> <span class="z-meta z-mapping z-key z-js">timeout</span><span class="z-punctuation z-separator z-key-value z-js">:</span> <span class="z-constant z-numeric z-integer z-decimal z-js">5000</span><span class="z-punctuation z-separator z-comma z-js">,</span> </span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-mapping z-js"> <span class="z-punctuation z-section z-block z-end z-js">}</span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> </span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"> <span class="z-meta z-function-call z-method z-js"><span class="z-variable z-other z-readwrite z-js">ruto</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-variable z-function z-js">send</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-meta z-string z-js"><span class="z-string z-quoted z-double z-js"><span class="z-punctuation z-definition z-string z-begin z-js">&quot;</span>https://example.com/parent-to-popup/topic1<span class="z-punctuation z-definition z-string z-end z-js">&quot;</span></span></span><span class="z-punctuation z-separator z-comma z-js">,</span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-variable z-other z-readwrite z-js">popup</span><span class="z-punctuation z-separator z-comma z-js">,</span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-variable z-other z-readwrite z-js">message</span><span class="z-punctuation z-separator z-comma z-js">,</span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-variable z-other z-readwrite z-js">options</span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-punctuation z-section z-group z-end z-js">)</span></span> </span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"> </span><span class="z-punctuation z-accessor z-js">.</span><span class="z-meta z-function-call z-method z-js"><span class="z-variable z-function z-js">then</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-meta z-function z-js"></span><span class="z-meta z-function z-declaration z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-meta z-binding z-name z-js"><span class="z-variable z-parameter z-function z-js">response</span></span><span class="z-punctuation z-section z-group z-end z-js">)</span> <span class="z-storage z-type z-function z-arrow z-js">=&gt;</span></span><span class="z-meta z-function z-js"> <span class="z-meta z-block z-js"><span class="z-punctuation z-section z-block z-begin z-js">{</span> </span></span></span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"><span class="z-meta z-function z-js"><span class="z-meta z-block z-js"> <span class="z-meta z-function-call z-method z-js"><span class="z-support z-type z-object z-console z-js">console</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-support z-function z-console z-js">log</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-variable z-other z-readwrite z-js">response</span><span class="z-punctuation z-section z-group z-end z-js">)</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> <span class="z-comment z-line z-double-slash z-js"><span class="z-punctuation z-definition z-comment z-js">//</span>Hello from popup </span></span></span></span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"><span class="z-meta z-function z-js"><span class="z-meta z-block z-js"> <span class="z-punctuation z-section z-block z-end z-js">}</span></span></span><span class="z-punctuation z-section z-group z-end z-js">)</span></span> </span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"> </span><span class="z-punctuation z-accessor z-js">.</span><span class="z-meta z-function-call z-method z-js"><span class="z-variable z-function z-js">catch</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-meta z-function z-js"></span><span class="z-meta z-function z-declaration z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-meta z-binding z-name z-js"><span class="z-variable z-parameter z-function z-js">error</span></span><span class="z-punctuation z-section z-group z-end z-js">)</span> <span class="z-storage z-type z-function z-arrow z-js">=&gt;</span></span><span class="z-meta z-function z-js"> <span class="z-meta z-block z-js"><span class="z-punctuation z-section z-block z-begin z-js">{</span> </span></span></span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"><span class="z-meta z-function z-js"><span class="z-meta z-block z-js"> <span class="z-meta z-function-call z-method z-js"><span class="z-support z-type z-object z-console z-js">console</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-support z-function z-console z-js">log</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-variable z-other z-readwrite z-js">error</span><span class="z-punctuation z-section z-group z-end z-js">)</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> </span></span></span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"><span class="z-meta z-function z-js"><span class="z-meta z-block z-js"> <span class="z-punctuation z-section z-block z-end z-js">}</span></span></span><span class="z-punctuation z-section z-group z-end z-js">)</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> </span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"></span></span><span class="z-meta z-tag z-script z-end z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;/</span><span class="z-entity z-name z-tag z-script z-html">script</span><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span> </span></code></pre> <p><strong>Popup:</strong></p> <pre data-lang="html" class="language-html z-code"><code class="language-html" data-lang="html"><span class="z-text z-html z-basic"><span class="z-meta z-tag z-script z-begin z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;</span><span class="z-entity z-name z-tag z-script z-html">script</span></span><span class="z-meta z-tag z-script z-begin z-html"> <span class="z-meta z-attribute-with-value z-html"><span class="z-entity z-other z-attribute-name z-html">src</span><span class="z-punctuation z-separator z-key-value z-html">=</span><span class="z-string z-quoted z-double z-html"><span class="z-punctuation z-definition z-string z-begin z-html">&quot;</span>https://cdn.jsdelivr.net/gh/rajnandan1/ruto/dist/ruto.min.js<span class="z-punctuation z-definition z-string z-end z-html">&quot;</span></span></span><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span><span class="z-meta z-tag z-script z-end z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;/</span><span class="z-entity z-name z-tag z-script z-html">script</span><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span> </span><span class="z-text z-html z-basic"><span class="z-meta z-tag z-script z-begin z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;</span><span class="z-entity z-name z-tag z-script z-html">script</span></span><span class="z-meta z-tag z-script z-begin z-html"><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span> </span><span class="z-text z-html z-basic"> <span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-variable z-other z-readwrite z-js">ruto</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-variable z-function z-js">receive</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-meta z-string z-js"><span class="z-string z-quoted z-double z-js"><span class="z-punctuation z-definition z-string z-begin z-js">&quot;</span>/parent-to-popup/topic1<span class="z-punctuation z-definition z-string z-end z-js">&quot;</span></span></span><span class="z-punctuation z-separator z-comma z-js">,</span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-support z-type z-object z-dom z-js">window</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-meta z-property z-object z-js">opener</span><span class="z-punctuation z-separator z-comma z-js">,</span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-meta z-function z-js"></span><span class="z-meta z-function z-declaration z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-meta z-binding z-name z-js"><span class="z-variable z-parameter z-function z-js">response</span></span><span class="z-punctuation z-separator z-parameter z-function z-js">,</span> <span class="z-meta z-binding z-name z-js"><span class="z-variable z-parameter z-function z-js">message</span></span><span class="z-punctuation z-section z-group z-end z-js">)</span> <span class="z-storage z-type z-function z-arrow z-js">=&gt;</span></span><span class="z-meta z-function z-js"> <span class="z-meta z-block z-js"><span class="z-punctuation z-section z-block z-begin z-js">{</span> </span></span></span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"><span class="z-meta z-function z-js"><span class="z-meta z-block z-js"> <span class="z-meta z-function-call z-method z-js"><span class="z-support z-type z-object z-console z-js">console</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-support z-function z-console z-js">log</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-variable z-other z-readwrite z-js">message</span><span class="z-punctuation z-section z-group z-end z-js">)</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> <span class="z-comment z-line z-double-slash z-js"><span class="z-punctuation z-definition z-comment z-js">//</span>Hello from parent </span></span></span></span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"><span class="z-meta z-function z-js"><span class="z-meta z-block z-js"> <span class="z-keyword z-control z-flow z-return z-js">return</span> <span class="z-meta z-function-call z-method z-js"><span class="z-variable z-other z-readwrite z-js">response</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-variable z-function z-js">send</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-meta z-string z-js"><span class="z-string z-quoted z-double z-js"><span class="z-punctuation z-definition z-string z-begin z-js">&quot;</span>Hello from popup<span class="z-punctuation z-definition z-string z-end z-js">&quot;</span></span></span><span class="z-punctuation z-section z-group z-end z-js">)</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> </span></span></span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"><span class="z-meta z-function z-js"><span class="z-meta z-block z-js"> <span class="z-punctuation z-section z-block z-end z-js">}</span></span></span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-punctuation z-section z-group z-end z-js">)</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> </span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"></span></span><span class="z-meta z-tag z-script z-end z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;/</span><span class="z-entity z-name z-tag z-script z-html">script</span><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span> </span></code></pre> <h3 id="popup-to-parent">Popup to Parent<a class="zola-anchor" href="#popup-to-parent" aria-label="Anchor link for: popup-to-parent" style="visibility: hidden;"></a> </h3> <p>Popup will send a message to parent, and expects a response from parent within 5 secs.</p> <p><strong>Parent:</strong></p> <pre data-lang="html" class="language-html z-code"><code class="language-html" data-lang="html"><span class="z-text z-html z-basic"><span class="z-meta z-tag z-script z-begin z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;</span><span class="z-entity z-name z-tag z-script z-html">script</span></span><span class="z-meta z-tag z-script z-begin z-html"> <span class="z-meta z-attribute-with-value z-html"><span class="z-entity z-other z-attribute-name z-html">src</span><span class="z-punctuation z-separator z-key-value z-html">=</span><span class="z-string z-quoted z-double z-html"><span class="z-punctuation z-definition z-string z-begin z-html">&quot;</span>https://cdn.jsdelivr.net/gh/rajnandan1/ruto/dist/ruto.min.js<span class="z-punctuation z-definition z-string z-end z-html">&quot;</span></span></span><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span><span class="z-meta z-tag z-script z-end z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;/</span><span class="z-entity z-name z-tag z-script z-html">script</span><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span> </span><span class="z-text z-html z-basic"><span class="z-meta z-tag z-script z-begin z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;</span><span class="z-entity z-name z-tag z-script z-html">script</span></span><span class="z-meta z-tag z-script z-begin z-html"><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span> </span><span class="z-text z-html z-basic"> <span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-storage z-type z-js">const</span> <span class="z-meta z-binding z-name z-js"><span class="z-variable z-other z-readwrite z-js">popup</span></span> <span class="z-keyword z-operator z-assignment z-js">=</span> <span class="z-meta z-function-call z-method z-js"><span class="z-support z-type z-object z-dom z-js">window</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-variable z-function z-js">open</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-meta z-string z-js"><span class="z-string z-quoted z-double z-js"><span class="z-punctuation z-definition z-string z-begin z-js">&quot;</span>https://example.com/popup.html<span class="z-punctuation z-definition z-string z-end z-js">&quot;</span></span></span><span class="z-punctuation z-separator z-comma z-js">,</span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-meta z-string z-js"><span class="z-string z-quoted z-double z-js"><span class="z-punctuation z-definition z-string z-begin z-js">&quot;</span>popup<span class="z-punctuation z-definition z-string z-end z-js">&quot;</span></span></span><span class="z-punctuation z-separator z-comma z-js">,</span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-meta z-string z-js"><span class="z-string z-quoted z-double z-js"><span class="z-punctuation z-definition z-string z-begin z-js">&quot;</span>width=600,height=400<span class="z-punctuation z-definition z-string z-end z-js">&quot;</span></span></span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-punctuation z-section z-group z-end z-js">)</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> </span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"> <span class="z-meta z-function-call z-method z-js"><span class="z-variable z-other z-readwrite z-js">ruto</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-variable z-function z-js">receive</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-meta z-string z-js"><span class="z-string z-quoted z-double z-js"><span class="z-punctuation z-definition z-string z-begin z-js">&quot;</span>/popup-to-parent/topic1<span class="z-punctuation z-definition z-string z-end z-js">&quot;</span></span></span><span class="z-punctuation z-separator z-comma z-js">,</span> <span class="z-variable z-other z-readwrite z-js">popup</span><span class="z-punctuation z-separator z-comma z-js">,</span> <span class="z-meta z-function z-js"></span><span class="z-meta z-function z-declaration z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-meta z-binding z-name z-js"><span class="z-variable z-parameter z-function z-js">response</span></span><span class="z-punctuation z-separator z-parameter z-function z-js">,</span> <span class="z-meta z-binding z-name z-js"><span class="z-variable z-parameter z-function z-js">message</span></span><span class="z-punctuation z-section z-group z-end z-js">)</span> <span class="z-storage z-type z-function z-arrow z-js">=&gt;</span></span><span class="z-meta z-function z-js"> <span class="z-meta z-block z-js"><span class="z-punctuation z-section z-block z-begin z-js">{</span> </span></span></span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"><span class="z-meta z-function z-js"><span class="z-meta z-block z-js"> <span class="z-meta z-function-call z-method z-js"><span class="z-support z-type z-object z-console z-js">console</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-support z-function z-console z-js">log</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-variable z-other z-readwrite z-js">message</span><span class="z-punctuation z-section z-group z-end z-js">)</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> <span class="z-comment z-line z-double-slash z-js"><span class="z-punctuation z-definition z-comment z-js">//</span>Hello from popup </span></span></span></span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"><span class="z-meta z-function z-js"><span class="z-meta z-block z-js"> <span class="z-keyword z-control z-flow z-return z-js">return</span> <span class="z-meta z-function-call z-method z-js"><span class="z-variable z-other z-readwrite z-js">response</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-variable z-function z-js">send</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-meta z-string z-js"><span class="z-string z-quoted z-double z-js"><span class="z-punctuation z-definition z-string z-begin z-js">&quot;</span>Hello from parent<span class="z-punctuation z-definition z-string z-end z-js">&quot;</span></span></span><span class="z-punctuation z-section z-group z-end z-js">)</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> </span></span></span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"><span class="z-meta z-function z-js"><span class="z-meta z-block z-js"> <span class="z-punctuation z-section z-block z-end z-js">}</span></span></span><span class="z-punctuation z-section z-group z-end z-js">)</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> </span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"></span></span><span class="z-meta z-tag z-script z-end z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;/</span><span class="z-entity z-name z-tag z-script z-html">script</span><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span> </span></code></pre> <p><strong>Popup:</strong></p> <pre data-lang="html" class="language-html z-code"><code class="language-html" data-lang="html"><span class="z-text z-html z-basic"><span class="z-meta z-tag z-script z-begin z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;</span><span class="z-entity z-name z-tag z-script z-html">script</span></span><span class="z-meta z-tag z-script z-begin z-html"> <span class="z-meta z-attribute-with-value z-html"><span class="z-entity z-other z-attribute-name z-html">src</span><span class="z-punctuation z-separator z-key-value z-html">=</span><span class="z-string z-quoted z-double z-html"><span class="z-punctuation z-definition z-string z-begin z-html">&quot;</span>https://cdn.jsdelivr.net/gh/rajnandan1/ruto/dist/ruto.min.js<span class="z-punctuation z-definition z-string z-end z-html">&quot;</span></span></span><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span><span class="z-meta z-tag z-script z-end z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;/</span><span class="z-entity z-name z-tag z-script z-html">script</span><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span> </span><span class="z-text z-html z-basic"><span class="z-meta z-tag z-script z-begin z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;</span><span class="z-entity z-name z-tag z-script z-html">script</span></span><span class="z-meta z-tag z-script z-begin z-html"><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span> </span><span class="z-text z-html z-basic"> <span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-storage z-type z-js">const</span> <span class="z-meta z-binding z-name z-js"><span class="z-variable z-other z-readwrite z-js">message</span></span> <span class="z-keyword z-operator z-assignment z-js">=</span> <span class="z-meta z-string z-js"><span class="z-string z-quoted z-double z-js"><span class="z-punctuation z-definition z-string z-begin z-js">&quot;</span>Hello from popup<span class="z-punctuation z-definition z-string z-end z-js">&quot;</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> </span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"> <span class="z-storage z-type z-js">const</span> <span class="z-meta z-binding z-name z-js"><span class="z-variable z-other z-readwrite z-js">options</span></span> <span class="z-keyword z-operator z-assignment z-js">=</span> <span class="z-meta z-mapping z-js"><span class="z-punctuation z-section z-block z-begin z-js">{</span> </span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-mapping z-js"> <span class="z-meta z-mapping z-key z-js">timeout</span><span class="z-punctuation z-separator z-key-value z-js">:</span> <span class="z-constant z-numeric z-integer z-decimal z-js">5000</span><span class="z-punctuation z-separator z-comma z-js">,</span> </span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-mapping z-js"> <span class="z-punctuation z-section z-block z-end z-js">}</span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> </span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"> <span class="z-meta z-function-call z-method z-js"><span class="z-variable z-other z-readwrite z-js">ruto</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-variable z-function z-js">send</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-meta z-string z-js"><span class="z-string z-quoted z-double z-js"><span class="z-punctuation z-definition z-string z-begin z-js">&quot;</span>https://example.com/popup-to-parent/topic1<span class="z-punctuation z-definition z-string z-end z-js">&quot;</span></span></span><span class="z-punctuation z-separator z-comma z-js">,</span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-support z-type z-object z-dom z-js">window</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-meta z-property z-object z-js">opener</span><span class="z-punctuation z-separator z-comma z-js">,</span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-variable z-other z-readwrite z-js">message</span><span class="z-punctuation z-separator z-comma z-js">,</span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-variable z-other z-readwrite z-js">options</span> </span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"> <span class="z-punctuation z-section z-group z-end z-js">)</span></span> </span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"> </span><span class="z-punctuation z-accessor z-js">.</span><span class="z-meta z-function-call z-method z-js"><span class="z-variable z-function z-js">then</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-meta z-function z-js"></span><span class="z-meta z-function z-declaration z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-meta z-binding z-name z-js"><span class="z-variable z-parameter z-function z-js">response</span></span><span class="z-punctuation z-section z-group z-end z-js">)</span> <span class="z-storage z-type z-function z-arrow z-js">=&gt;</span></span><span class="z-meta z-function z-js"> <span class="z-meta z-block z-js"><span class="z-punctuation z-section z-block z-begin z-js">{</span> </span></span></span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"><span class="z-meta z-function z-js"><span class="z-meta z-block z-js"> <span class="z-meta z-function-call z-method z-js"><span class="z-support z-type z-object z-console z-js">console</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-support z-function z-console z-js">log</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-variable z-other z-readwrite z-js">response</span><span class="z-punctuation z-section z-group z-end z-js">)</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> <span class="z-comment z-line z-double-slash z-js"><span class="z-punctuation z-definition z-comment z-js">//</span>Hello from parent </span></span></span></span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"><span class="z-meta z-function z-js"><span class="z-meta z-block z-js"> <span class="z-punctuation z-section z-block z-end z-js">}</span></span></span><span class="z-punctuation z-section z-group z-end z-js">)</span></span> </span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"> </span><span class="z-punctuation z-accessor z-js">.</span><span class="z-meta z-function-call z-method z-js"><span class="z-variable z-function z-js">catch</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-meta z-function z-js"></span><span class="z-meta z-function z-declaration z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-meta z-binding z-name z-js"><span class="z-variable z-parameter z-function z-js">error</span></span><span class="z-punctuation z-section z-group z-end z-js">)</span> <span class="z-storage z-type z-function z-arrow z-js">=&gt;</span></span><span class="z-meta z-function z-js"> <span class="z-meta z-block z-js"><span class="z-punctuation z-section z-block z-begin z-js">{</span> </span></span></span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"><span class="z-meta z-function z-js"><span class="z-meta z-block z-js"> <span class="z-meta z-function-call z-method z-js"><span class="z-support z-type z-object z-console z-js">console</span><span class="z-punctuation z-accessor z-js">.</span><span class="z-support z-function z-console z-js">log</span><span class="z-meta z-group z-js"><span class="z-punctuation z-section z-group z-begin z-js">(</span><span class="z-variable z-other z-readwrite z-js">error</span><span class="z-punctuation z-section z-group z-end z-js">)</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> </span></span></span></span></span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"><span class="z-meta z-function-call z-method z-js"><span class="z-meta z-group z-js"><span class="z-meta z-function z-js"><span class="z-meta z-block z-js"> <span class="z-punctuation z-section z-block z-end z-js">}</span></span></span><span class="z-punctuation z-section z-group z-end z-js">)</span></span></span><span class="z-punctuation z-terminator z-statement z-js">;</span> </span></span></span><span class="z-text z-html z-basic"><span class="z-source z-js z-embedded z-html"><span class="z-source z-js"></span></span><span class="z-meta z-tag z-script z-end z-html"><span class="z-punctuation z-definition z-tag z-begin z-html">&lt;/</span><span class="z-entity z-name z-tag z-script z-html">script</span><span class="z-punctuation z-definition z-tag z-end z-html">&gt;</span></span> </span></code></pre>