Case Study: Fynd Uses Kafka and Rockset to Respond to E-Commerce Consumer Behavior in Real Time
March 19, 2019
Fynd is an online to offline (O2O) fashion e-commerce portal that brings in-store fashion products from retail brands to an online audience. Fynd pulls real-time streams of inventory data from over 9,000 stores in India to provide its 17 million customers up-to-date information on the latest offers and trends in fashion. Data and technology are at the heart of Fynd’s business.
Real Time Is Critical in Retail E-Commerce
As a retail e-commerce company, Fynd’s business is predicated upon its ability to respond to consumer behavior as it happens. Fynd is constantly monitoring transactions and activity on its platform to uncover issues and trends in orders, inventory management, and security. Fynd only has a very short amount of time in which to identify these situations before the opportunity to respond is lost.
Fynd works in concert with their retail brand partners to put on limited-time sales that could last a week, several days, or even minutes. Fynd experiences significant traffic during these sales. A 2-minute sale could see a million concurrent users on Fynd’s platform, and Fynd needs to know everything about the sale while it is going on.
Fynd’s marketing team is an analytics powerhouse, and asks a host of questions about their sales. How many orders are coming in? What are the top-selling brands, products, and price ranges? Are there geographic areas that are outperforming others? In which demographics is the sale performing best? And they need answers in real time to adjust their marketing tactics to optimize Fynd’s sales performance.
Live metrics are also very important to the team in assessing where they stand relative to sales targets. A retail brand may have predetermined a certain volume of product they wish to sell for a discount, for example, and Fynd needs to react to sales conditions in real time in light of these targets.
From an operations perspective, Fynd tracks metrics like the number of visitors on the platform, orders coming from different channels, and the response times of critical systems, constantly refreshing live dashboards with these metrics. Fynd has to immediately detect unusual events. Is there an issue with the site that is causing a problem for the consumer, or is there is a consumer on the site causing a problem for Fynd? Fynd needs to know if the number of orders coming in is abnormally high or low, for instance, which could be symptomatic of fraud or a problem with the payments backend, respectively.
30 Minutes Is Too Long
To power their business, Fynd collects data on many types of events from its mobile and web applications. During campaigns, Fynd’s users could generate 30 million events per day, and all the data that is produced is streamed into Kafka.
Fynd would prepare the data and load it into one of several analytics platforms in the cloud, so that it could be queried to support marketing decisions. But that process required a minimum of 30 minutes—too long for an online business like Fynd. Any consumer behavior discovered through this flow would be long gone before Fynd could respond.
Fast Queries on Real-Time Streams in Kafka
Fynd’s technical team turned to Rockset to reduce the time it took from data to insight. Instead of loading the data periodically from Kafka, Rockset connects to Kafka to continuously sync new data.
Fynd’s real-time JSON event streams are automatically ingested and schematized without any manual intervention, so Fynd can perform SQL queries right away in Rockset. Another difference is the improved performance Fynd experiences on their queries, as Rockset fully indexes all of Fynd’s data to deliver millisecond-latency SQL.
With Rockset as part of the data flow, Fynd developed a serverless microservice to keep tabs on their key metrics. Using AWS Lambda functions in conjunction with Rockset’s client libraries, the technical team created a feature that fires off a query to Rockset whenever an endpoint is called. Fynd can now refresh metrics and live dashboards multiple times a minute in a lightweight, serverless manner.
Better Decisions, More Scalable Systems at Fynd
By using Rockset on the critical path, Fynd can now obtain immediate insight into what consumers are doing on their platform. And they can react more quickly and more effectively, making better decisions to maximize campaign results, than before.
The new flow also eliminates much of the management and monitoring of the data platform. There are no servers to provision when building on Rockset, no infrastructure or data warehouse administration, and no requirement to prepare and load data as Rockset continuously ingests new data. This frees up the technical team to work on tasks with more direct revenue impact.
“We need to carefully monitor our growth in real-time. Is a certain product suddenly selling more? Is there a fraudulent transaction? We easily generate 20-30 million events per day, all captured in Kafka streams. Our applications query the data every few seconds. By sending our raw event data directly from Kafka to Rockset, we save a lot of time and energy. We track over 40 metrics in real time and constantly take immediate actions,” says Amboj Goyal, Principal Engineer at Fynd
In an attempt to get to the data more quickly, some marketing queries are bypassing the analytical systems and hitting the operational databases today, which is not ideal. Amboj intends to offload these queries to Rockset, which is better suited for such workloads, and track even more metrics using Rockset in the near future. Amboj also looks forward to scaling Fynd’s data platform with Rockset to support Fynd’s growth.