Visualizing real-time data streams

The team that built Apache Kafka at LinkedIn founded Confluent as a streaming data platform powered by Kafka, which was able to accommodate over 1 trillion messages per day. Confluent focuses on helping companies get easy access to enterprise data as real-time streams. Confluent asked us to help them design their reporting interface, which would let users see where the bottlenecks and problems are in the messages they publish to Kafka.

The Challenge

A big challenge was to visually represent the massive amount of data while still being legible and usable. Kafka can accommodate over 1 trillion messages a day, so our unit for time measurement was by the minute. Not only did we need to display this data, but we needed to break it down into several factors, such as messages produced, duplicated, lost, and lag time.

Onsite Meeting

We went to Confluent’s office in Palo Alto, CA to talk with them in person and to better understand their need. Together, we brainstormed and discussed ways to display their data.

Charts, Charts, Charts

We worked in wireframes so that we could focus on interactions and charts without getting caught up in colors or styling. In our search for the most appropriate data visualizations, we explored different kinds of charts and how they would function in context of one another. We drew inspiration from our past experience, best practices, charting libraries, and examples in articles or textbooks. We even took ideas from baseball stats!

Two Concepts

From the same set of wireframes, different designers can interpret them in different ways. To give Confluent some options regarding look and feel, two of our designers each created a concept containing a dark theme and a light theme.

Visual Design Considerations

As we designed, we realized that the red and green of a typical heat map would not work for colorblind users. Instead of relying on color, we relied on opacity and simplified the heat map so that only the problem areas showed.

Final Design



As we worked with Confluent to design their reporting interface, the biggest challenge was visually representing the data in a legible, useful way. Through explorations and iterations, we designed a dashboard and detail page that is not only aesthetically fresh and appealing, but also usable.