How kafkaflow supercharged FARFETCH’s event-driven architecture

By Filipe Esch and Guilherme Ferreira

FARFETCH relies on an event-driven architecture to handle more than 150k messages per second, only in one region. Leveraging Apache Kafka, this architecture serves as the backbone of the operations. But autonomy comes at a price in a distributed system like this. Having many of our services developed in .NET, we could see many different Apache Kafka adapters being developed. Even using Confluent Kafka Client for .NET, the available feature set fell short, leading to multiple teams solving the same problems. This is why KafkaFlow has born, intending to build a standard and unified approach to working with Kafka for .NET Developers. KafkaFlow is designed to construct event-driven applications atop Kafka using .NET. KafkaFlow not only offers a consistent, reliable, and maintainable approach to application development but also enhances organisational efficiency. Due to its success, FARFETCH open-sourced KafkaFlow. This case study delves into the challenges FARFETCH faced and the benefits obtained by the global adoption of KafkaFlow.

The Challenge

As we began to integrate Apache Kafka as a key element in our architecture, we quickly observed the potential risks associated with the absence of standardization. If we do not come up with a standardized approach to connecting with Apache Kafka, it would result in disparities in application development, which would turn into elevated maintenance costs. This could hamper the productivity and efficiency of our entire team, affecting everything from simple tasks such as patching to more complex aspects such as feature development and knowledge sharing.

Additionally, we identified the possibility of fragmentation among our internal Kafka libraries, which could lead to complications in reusing code and negatively impact maintainability. We observed interesting features implemented in certain libraries but omitted in others. This inconsistency was mirrored in monitoring, tracing, and telemetry, where different solutions were used across libraries. Some libraries even employed incompatible message notation, creating unnecessary work facilitating basic application communication. In Kafka’s earlier versions, the lack of support for message headers necessitated building this functionality into our libraries. This presented another problem, as each library adopted different methods to achieve this. In essence, it needed a quick reaction before it could turn into a chaotic situation!

Solution

To address these challenges, we designed KafkaFlow with a robust middleware system that empowers developers by giving them full control over message consumption and production. KafkaFlow’s middleware architecture gives the client the flexibility to support diverse message notations, thus enabling seamless integration with different libraries.

The middleware system in KafkaFlow serves as an intermediate layer that processes messages and permits message enrichment and transformation (a transformation example is message deserialisation). This feature is especially beneficial in creating support for reusable code which can communicate with various libraries. It also simplifies common operations such as message compression/decompression, envelope creation for messages, and custom serialisation formats.

Moreover, KafkaFlow’s middleware extends its capabilities to vital aspects such as monitoring, tracing, and logging. These features are critical in ensuring the codebase’s maintainability and keeping the system’s performance optimal. It provides a coherent and structured way of interfacing with Apache Kafka, eliminating inconsistencies in application development and reducing the overall maintenance cost.

KafkaFlow is not only designed to be extremely extensible but also has a reach feature set, where we can find features like:

• Multi-topic Consumers.
• ApacheAvro, ProtoBuf, and JSON serialisation algorithms.
• Schema Registry support.
• Compression using native Confluent Kafka client compression or compressor middleware.
• Automatic store offset, avoiding message loss.
• Admin Web API and Dashboard UI that allows pause, resume, and restart consumers, change workers count, and rewind offsets, all at runtime.

Initially, the library was adopted for a few critical services. After successful testing and validation, it reached a maturity where we feel comfortable extending it to all .NET services that depend on Apache Kafka.

Product Catalog

Managing the product catalogue is a crucial aspect of FARFETCH operations. KafkaFlow is heavily used to support this process, which encompasses inventory tracking and publication of products. This integration ensures that all aspects of the product catalogue are consistently maintained and up-to-date, as all the involved services use KafkaFlow to produce and consume messages that orchestrate this process efficiently.

Fig.1 — Catalogue management — areas of responsibility

Two KafkaFlow features are a key part of the Product Catalog management: Batch Consumption and Multi-threaded consumers with message order guarantee.

KafkaFlow’s support for batch consumption plays a vital role in optimising the performance of product catalogue management. With batch consumption, messages from Kafka can be efficiently processed in groups rather than individually, reducing the overhead associated with processing each message separately. This approach enables handling large volumes of catalogue updates and inventory changes easily, ensuring that the product catalogue remains current and accurate at all times. As a result, batch consumption contributes to the overall performance and responsiveness of the systems.

Implementing a feature like Batch Consuming that preserves order it’s a massive project. However, with KafkaFlow, each team can adopt using a couple of lines of code in a few minutes.

Fig.2 — KafkaFlow Batch Consumer configuration

Parallelism is another key feature of KafkaFlow that supports the performance and resource efficiency of product catalogue management. Parallelism support allows the company to distribute the processing workload across multiple consumer instances, utilising the available infrastructure resources. By evenly distributing tasks and leveraging the full potential of its server instances, FARFETCH can process and update its product catalogue swiftly while minimising resource consumption. This efficient use of infrastructure resources reduces operational costs and enables scaling the catalogue management processes seamlessly as the business grows.

The two features mentioned are an investment that not only supports the processes around the product catalogue but also other key business operations. Due to their complexity, the return on investment in building them as part of KafkaFlow is massive.

Results

KafkaFlow revolutionised those applications, delivering substantial improvements in productivity and effectiveness. The key results included:

1. Increased efficiency: KafkaFlow reduced the time spent on development, maintenance, and troubleshooting, allowing developers to focus on adding value to the business.
2. Improved reliability: The library’s robust design ensured minimal downtime and consistently high performance, even under heavy loads.
3. Stronger collaboration: By adopting a shared toolset and approach, team members were better equipped to collaborate and coordinate their efforts.
4. Security: Having a framework-designated team reduces the lead time to react to security issues and increases rollout efficiency.
5. Standardization: KafkaFlow enabled a standardised way to build event-driven applications. The consistency across applications increases efficiency and accelerates developer onboarding.

FARFETCH’s journey with KafkaFlow provided valuable insights into the benefits of developing custom, purpose-built tools to address unique challenges. Some key takeaways include:

1. Technology alignment: Common purpose frameworks allow knowledge sharing and economies of scale.
2. Incremental adoption: Gradually integrating new tools into existing systems can mitigate risks and ensure a smooth transition.
3. Fostering collaboration: A unified approach to application development promotes better communication and teamwork among developers.

Open-Source

KafkaFlow was developed with Open-Source in mind. Even though the first steps were in an inner-source way, all the design decisions were conscious of the role that this framework could play in the community.

We expected that moving to Open Source would be a multiplier of the benefits of collaboration. Nowadays, KafkaFlow keeps evolving, driven by FARFETCH needs and community contribution.

Conclusion

KafkaFlow’s implementation showcased the power of leveraging crafted tools in addressing common organizational challenges. By fostering collaboration, KafkaFlow enabled FARFETCH to maximise the potential of its event-driven architecture, resulting in significant gains in productivity and effectiveness. The success of KafkaFlow demonstrates the importance of Open-Source practices and inspires other organisations facing similar challenges.

Originally published at https://www.farfetchtechblog.com on August 28th, 2023.