As companies become more data and software-driven, there is a push by technology leadership to reduce the latency gap between the time data is produced and when it is actioned. We see this in many industries and use cases:All of these examples have real consequences if data is delayed, so it’s important to reduce the total time to action to stay competitive.In this article, we’ll focus on a use case that brings billions of data points and thousands of companies together in milliseconds to make instant decisions: real-time online ad bidding.The old advertising saying is that “half the money I spend on advertising is wasted; the trouble is I don’t know which half.” But data is a lot easier to come by in the digital age. For example, ad exchanges can anonymously share that visitors on a given web property have recently visited both StubHub and Lionel Messi’s Instagram page. A paid ad for an Inter Miami jersey on that page is money well spent and is much more likely to create a happy customer than an untargeted ad for a jersey from a different team or sport.Interactions like this — the decisions to evaluate, buy and place ads — happen in milliseconds, millions of times a day. Critical to these transactions is the free flow of data for timely actions, often facilitated by real-time databases and streaming data platforms. This article unveils how Demand Side Platforms (DSPs), in particular, can implement a solution leveraging the capabilities of Redpanda (a streaming data platform) and Aerospike (a real-time database) to achieve success in the fast-paced environment of real-time bidding.Aerospike is a real-time database that ingests, stores and retrieves data, handling millions of transactions per second (TPS) throughput with sub-millisecond latency. Event streams are one of the frequent sources of data ingested into an Aerospike database. Aerospike can ingest events directly, or through its connectors, to streaming data platforms like Redpanda, Apache Kafka, Apache Pulsar and others.It’s common to ingest data into Aerospike from a streaming topic, run several evaluations of that data and publish data to another topic, all in milliseconds. In ad tech, one key use case for Aerospike is as a user profile store. This store holds extensive anonymized information about individual users, from preferences to past interactions.You can think of Redpanda as a rebuilt Kafka. Kafka, one of the most popular streaming platforms, ensures seamless communication among various parts of the online ecosystem. Redpanda uses the Kafka API to ensure compatibility with the existing ecosystem but is rewritten from the ground up in C++ to maximize modern hardware utilization. One of Redpanda’s selling points is its ability to reliably handle large spikes in volume, supporting up to multiple gigabytes (GB) per second on average.To understand how Dsps fit into the ad tech landscape, we need a glimpse into real-time bidding (RTB). In RTB, advertisers compete to display their ads on websites in real time, which entails a complex engagement between supply-side platforms (SSPs), DSPs, websites, and ad exchanges. DSPs play a crucial role by representing advertisers and making split-second decisions to bid on ad slots. DSPs must differentiate themselves by swiftly processing data, targeting the right audience and optimizing bids to win the auction.Figure 1: Real-time bidding overview In the interconnected world of DSPs, multiple key data components come into play: user profiles, available ad slots, historical data, and ad creatives. DSPs need to rapidly create profiles of potential users based on available data, assess ad slot opportunities from SSP or ad exchanges, and craft bids that align with user preferences and active campaigns. This process requires efficient data movement and rapid decision-making.This is where Redpanda and Aerospike come in. Redpanda provides a highly performant streaming engine, enabling lumpy spikes in data volumes to move through the DSP’s internal environment and across the rest of the Ad Tech ecosystem in milliseconds. Aerospike serves as the data historian, delivering speedy retrieval of data lookups such as users, devices and sessions. It swiftly retrieves user profiles and past interactions, allowing DSPs to craft bids tailored to individual preferences.Figure 2: Real-time data architecture for a DSPAs illustrated in Figure 2, DSPs incorporate information from ad exchanges and the publisher side, such as available ad formats and anonymized user interests and geolocation data. Some DSPs budget just 10 milliseconds to load all this data to inform their bidding process. Aerospike can ingest data in under a millisecond and will typically connect directly to an ad exchange’s API.There are many examples where it makes more sense to decouple the data producer from the data consumer. In these cases Aerospike can use its Kafka Connector to publish the ad exchange data to a Redpanda topic, enabling multiple services within the DSP to subscribe to the topic. A pricing optimization service, for example, could augment new real-time data with historical information about targeted users to help predict the ad spend for upcoming bids.Redpanda provides a Kafka API that consistently delivers p50 latency as low as 5 milliseconds. Some of the DSP’s external data will go directly onto a Redpanda topic, such as data on impressions and bidding results. Aerospike can also be a subscriber to any of these topics, as it extends the historical database.DSPs harness the Kafka API to swiftly process incoming data, assess ad opportunities and generate bids. The real-time bidding environment demands speed, accuracy and the ability to handle immense data flows. The combination of Aerospike and Redpanda provides both real-time data flow and the nearly instantaneous retrieval of past data to help DSPs make informed, tailored bids. This movement and integration of data ensures that bid requests, ad opportunities and decisions flow seamlessly, enabling timely ad placements.In the dynamic world of real-time bidding, Demand Side Platforms (DSPs) depend on powerful, modern platforms and efficient tooling to move and synthesize disparate data in mere milliseconds, ultimately making the best bid for their clients. In one such architecture, Redpanda’s performant Kafka API synchronizes seamlessly with Aerospike’s lightning-fast retrieval of user profiles, ensuring that DSPs are armed with all the information they need to craft real-time bids that are highly personalized for users. With these platforms in their arsenal, DSPs can navigate the intricate real-time bidding ecosystem with agility and precision, paving the way for successful ad placements and enriched user experiences.Community created roadmaps, articles, resources and journeys fordevelopers to help you choose your path and grow in your career.