Picnic Modernizes Data Architecture with Confluent Cloud

Picnic, Europe's rapidly expanding online-only supermarket, operating in France, the Netherlands, and Germany, has revolutionized its data architecture using Confluent for streaming analytics. Committed to data-driven strategies, Picnic guarantees the lowest prices, processing over 300 million events weekly from customer applications and internal systems to fuel predictive analytics within their data warehouse. Faced with exponential growth, Picnic sought a more reliable and high-performing data streaming platform to enhance its streaming analytics capabilities.

Picnic encountered challenges with its existing AWS Kinesis data pipelines, including limited data storage, extensive custom tooling requirements, and the need for exactly-once semantics. Addressing these issues was crucial for improving customer behavior analytics and streamlining internal processes. The company has two primary data pipelines, for customer-facing application and internal backend systems. The customer-facing application collects data on user behavior, which is analyzed to improve product recommendations and enhance the application UI. The second data pipeline processes data for internal backend systems, which helps with data warehouse, payment processing, and product availability status.

Confluent Cloud emerged as the ideal solution, built upon Apache Kafka, enabling Picnic to redesign its data pipelines for simplified internal services and more efficient processing of customer application data. The transition to Confluent yielded several key benefits, including infinite storage capabilities, a rich ecosystem of pre-built, fully managed connectors, and the realization of exactly-once semantics.

Picnic leverages connectors for RabbitMQ to seamlessly forward data into Confluent Cloud Kafka topics. Fully managed sink connectors then load data into Snowflake and Amazon S3 for in-depth analysis by data science teams. Confluent’s Data Preview feature further streamlines the process, allowing for iterative testing of connector outputs before production deployment. This optimized setup has led to enhanced scalability and streamlined data streams, complemented by Confluent’s managed connectors for improved monitoring of APIs for solutions like Prometheus. The company plans to leverage ksqlDB for new streaming analytics use cases like powering real-time reporting dashboards, for streaming ETL, and to enable things like real-time recommendations based on user behavior. Picnic also wants to improve data delivery to and from its automated fulfillment center and create a self-service platform for data delivery via automated deployment systems for Confluent-based pipelines.

Value Results:

• Reduced infrastructure costs by 40%.

• Simplified IT architecture, reducing the number of managed services and the overall maintenance burden.

• Improved infrastructure monitoring for better SLAs and prevention of data loss.

• Future-proof data architecture, capable of scaling with Picnic's continued growth and evolving needs.