Tech
June 9

Building scalable and reliable data pipelines using Debezium and Kafka

Data pipelines are used to perform data integration, which is the process of bringing together data from multiple sources to provide a complete and accurate dataset for business intelligence (BI), data analysis, and other applications. In this talk, we will cover this process.
Talk abstract

Data is being generated at an unprecedented rate, making it essential to have efficient and robust data pipelines to process and store data. In this talk, we will explore how to build scalable and reliable data pipelines using Debezium and Kafka.

Debezium is an open-source project that provides a platform for change data capture (CDC) from databases and streams the changes to Apache Kafka. This allows us to monitor changes to data in real-time and feed it into various applications.

Kafka on the other hand, is a distributed, high-performance, and scalable messaging system that acts as the backbone of modern data pipelines. With its pub-sub model, Kafka provides a mechanism for reliable data ingestion and distribution.

In this talk, we will cover the following topics:

  • Introduction to Debezium and Kafka
  • Setting up Debezium to capture changes from databases
  • Integrating DBs to Debezium with Kafka for real-time data streaming
  • Best practices for designing and deploying data pipelines using these technologies

Attendees will learn how to build efficient data pipelines that can handle large amounts of data and how to ensure the reliability and scalability of these pipelines. Whether you are a data engineer, developer, or architect, this talk is for you.

Join us for a deep dive into the world of Debezium and Kafka and learn how to build data pipelines that can handle the most demanding use cases.

Karan Thakur
Site Reliability Engineer, Moss

Related talks

Tech
June 9
Automate R&D workflows using serverless functions
As the cloud-native environment becomes increasingly complex and dynamic, it presents numerous challenges for platform teams. Managing software development manually is difficult and prone to human error and security issues, making automation essential for reducing workload and increasing security.
Michael Yuan
Founder & CEO, Second State
sponsored
Tech
June 8
Abstracting Kubernetes and setting standards with internal developer portals
Developers find it difficult to manage Kubernetes, let alone understand it when troubleshooting or developing in a shift left world. Internal developer portals can be used to abstract and whitelist K8s data and set standards through scorecards.
Suzanne Daniels
Developer Relations Lead, Port
sponsored
Tech
June 9
The current state and future of Internal Developer Portals
This session will cover how an internal developer portal and platform helps organizations, the current state of the internal developer portal product category, and the future of internal developer portals.
Seth Demsey
Co-Founder, configure8
sponsored
Tech
June 9
How to deploy Kotlin applications to Kubernetes without Dockerfiles
In this talk Lothar shares how to create docker images with Kotlin and Gradle only. Lothar will showcase a ktor application as well as create and deploy a Docker image that contains the application to a Kubernetes cluster. The deployment happens on the command line only - every code change will trigger an instant deployment of a newly created Docker image to a Kubernetes cluster.
Lothar Schulz
Head of Engineering, Miro
sponsored