How Kafka Connect Works: Integrating Data Between Systems

相关文章推荐

深情的硬盘 · 易二零水网· 1 年前 ·

成熟的皮蛋 · 熊出没探险日记：五大不合理的剧情，全部都是赵 ...· 1 年前 ·

博学的钢笔 · 无间行者之生死潜行迅雷下载_无间行者之生死潜 ...· 1 年前 ·

</noscript><div id="___gatsby"><div style="outline:none" tabindex="-1" id="gatsby-focus-wrapper"><i class="style-module--scrollIndicator--AGKy6"/><header class="style-module--header--aS7Z-"><div class="style-module--container--D9WQC"><a class="style-module--logo--Mp1H9" data-test-id="link" href="/"><img src="/images/developer-logo.svg" alt=""/></a><div class="style-module--navMobile--ucY9a"><a role="button" title="Get Started Free" class="Button-module--button--A-Z1p Button-module--island--2yOxk style-module--quickStartNavButton--u4TJC" data-test-id="link" href="https://www.confluent.io/get-started/">Get Started Free</a><button type="button" class="Button-module--button--A-Z1p style-module--mobileMenu---yHIt"><span/><span/><span/></button><nav class="style-module--mobileNavigation--1ZDxw" data-swiftype-index="false"><a role="button" title="Get Started Free" class="Button-module--button--A-Z1p Button-module--white--M+sIx style-module--quickStart--1W+K8" data-test-id="link" href="https://www.confluent.io/get-started/">Get Started Free</a><div class="style-module--accordionItemWrapper--Ds610 style-module--collapsed--ydSG7 style-module--mobileAccordionItem--m9OUc"><div class="style-module--accordionItem--Bex3L style-module--accordionItem--dbe+-"><div class="style-module--accordionTitle--gPya- style-module--titleClassName--CWu6r">Courses<div class="IconArrow-module--arrow--ULhik IconArrow-module--down--49+OW IconArrow-module--md--9BJoY IconArrow-module--theme_white--kOiXH style-module--indicator--FKh5h"/></div><div class="style-module--accordionPanel--NujUc"><ul class="style-module--menuGroup--rr-S+"><li class="style-module--navItemHeaderContainer--vxHRS"><div class="style-module--navItemGroupWrapper--utjJn style-module--roundTopBorderForTablets--SlQz4"><div><div class="style-module--titleWithOutMarginTop--QErYg">What are the courses?</div><span class="style-module--subline--cWWdf">Video courses covering Apache Kafka basics, advanced concepts, setup and use cases, and everything in between.</span><a class="style-module--navItemGroupCta--YzEUp" data-test-id="link" href="/courses/">Learning Pathways (24)<span/></a></div></div></li><li class="style-module--navItemLinearGroupContainer--E8b-c"><ul class="style-module--navItemGroup--CtPAh"><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/courses/microservices/from-monoliths-to-microservices/"><div class="style-module--navItem--1kjpc"><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys"><span class="style-module--indicator--WTN0y">NEW</span>Designing Event-Driven Microservices</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/courses/apache-flink/intro/"><div class="style-module--navItem--1kjpc"><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys"><span class="style-module--indicator--WTN0y">NEW</span>Apache Flink® 101</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/courses/flink-java/overview/"><div class="style-module--navItem--1kjpc"><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys"><span class="style-module--indicator--WTN0y">NEW</span>Building Flink® Apps in Java</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/courses/hybrid-cloud/intro/"><div class="style-module--navItem--1kjpc"><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys"><span class="style-module--indicator--WTN0y">NEW</span>Hybrid and Multicloud Architecture</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/courses/data-streaming-systems/overview/"><div class="style-module--navItem--1kjpc"><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys"><span class="style-module--indicator--WTN0y">NEW</span>Mastering Production Data Streaming Systems with Apache Kafka®</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://developer.confluent.io/courses/apache-kafka/events/"><div class="style-module--navItem--1kjpc"><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Kafka® 101</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://developer.confluent.io/courses/kafka-connect/intro/"><div class="style-module--navItem--1kjpc"><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Kafka® Connect 101</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://developer.confluent.io/courses/kafka-streams/get-started/"><div class="style-module--navItem--1kjpc"><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Kafka Streams 101</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://developer.confluent.io/courses/schema-registry/key-concepts"><div class="style-module--navItem--1kjpc"><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Schema Registry 101</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://developer.confluent.io/courses/ksqldb/intro/"><div class="style-module--navItem--1kjpc"><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">ksqlDB 101</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://developer.confluent.io/courses/data-mesh/intro/"><div class="style-module--navItem--1kjpc"><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Data Mesh 101</h4></div></div></a><span/></li></ul></li><li class="style-module--navItemNonLinearGroupContainer--0LVfk style-module--roundBottomBorderForTablets--bEt6C"><ul class="style-module--navItemGroup--CtPAh"><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/courses/microservices/from-monoliths-to-microservices/"><div class="style-module--navItem--1kjpc"><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys"><span class="style-module--indicator--WTN0y">NEW</span>Designing Event-Driven Microservices</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/courses/apache-flink/intro/"><div class="style-module--navItem--1kjpc"><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys"><span class="style-module--indicator--WTN0y">NEW</span>Apache Flink® 101</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/courses/flink-java/overview/"><div class="style-module--navItem--1kjpc"><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys"><span class="style-module--indicator--WTN0y">NEW</span>Building Flink® Apps in Java</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/courses/hybrid-cloud/intro/"><div class="style-module--navItem--1kjpc"><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys"><span class="style-module--indicator--WTN0y">NEW</span>Hybrid and Multicloud Architecture</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/courses/data-streaming-systems/overview/"><div class="style-module--navItem--1kjpc"><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys"><span class="style-module--indicator--WTN0y">NEW</span>Mastering Production Data Streaming Systems with Apache Kafka®</h4></div></div></a><span/></li></ul><ul class="style-module--navItemGroup--CtPAh"><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://developer.confluent.io/courses/apache-kafka/events/"><div class="style-module--navItem--1kjpc"><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Kafka® 101</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://developer.confluent.io/courses/kafka-connect/intro/"><div class="style-module--navItem--1kjpc"><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Kafka® Connect 101</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://developer.confluent.io/courses/kafka-streams/get-started/"><div class="style-module--navItem--1kjpc"><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Kafka Streams 101</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://developer.confluent.io/courses/schema-registry/key-concepts"><div class="style-module--navItem--1kjpc"><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Schema Registry 101</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://developer.confluent.io/courses/ksqldb/intro/"><div class="style-module--navItem--1kjpc"><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">ksqlDB 101</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://developer.confluent.io/courses/data-mesh/intro/"><div class="style-module--navItem--1kjpc"><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Data Mesh 101</h4></div></div></a><span/></li></ul></li></ul></div></div></div><div class="style-module--accordionItemWrapper--Ds610 style-module--collapsed--ydSG7 style-module--mobileAccordionItem--m9OUc"><div class="style-module--accordionItem--Bex3L style-module--accordionItem--dbe+-"><div class="style-module--accordionTitle--gPya- style-module--titleClassName--CWu6r">Learn<div class="IconArrow-module--arrow--ULhik IconArrow-module--down--49+OW IconArrow-module--md--9BJoY IconArrow-module--theme_white--kOiXH style-module--indicator--FKh5h"/></div><div class="style-module--accordionPanel--NujUc"><ul class="style-module--menuGroup--rr-S+"><li class="style-module--navItemHeaderContainer--vxHRS"><div class="style-module--navItemGroupWrapper--utjJn style-module--roundTopBorderForTablets--SlQz4"><div><div class="style-module--titleWithOutMarginTop--QErYg">Pick your learning path</div><span class="style-module--subline--cWWdf">A wide range of resources to get you started</span><a class="style-module--navItemGroupCta--YzEUp" data-test-id="link" href="https://developer.confluent.io/courses">Start Learning<span/></a></div></div></li><li class="style-module--navItemLinearGroupContainer--E8b-c"><ul class="style-module--navItemGroup--CtPAh"><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/learn/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Articles</h4><p class="style-module--subline--cWWdf">Deep-Dives into key concepts</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/patterns/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Patterns</h4><p class="style-module--subline--cWWdf">Architectures for event streaming</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/faq/apache-kafka/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">FAQs</h4><p class="style-module--subline--cWWdf">Q &amp; A about Kafka® and its ecosystem</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://www.confluent.io/blog/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Blog</h4><p class="style-module--subline--cWWdf">The Confluent blog</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/learn-more/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys"><span class="style-module--indicator--WTN0y">NEW</span>Learn More</h4><p class="style-module--subline--cWWdf">Other ways to learn</p></div></div></a><span/></li></ul></li><li class="style-module--navItemNonLinearGroupContainer--0LVfk style-module--roundBottomBorderForTablets--bEt6C"><ul class="style-module--navItemGroup--CtPAh"><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/learn/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Articles</h4><p class="style-module--subline--cWWdf">Deep-Dives into key concepts</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/patterns/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Patterns</h4><p class="style-module--subline--cWWdf">Architectures for event streaming</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/faq/apache-kafka/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">FAQs</h4><p class="style-module--subline--cWWdf">Q &amp; A about Kafka® and its ecosystem</p></div></div></a><span/></li></ul><ul class="style-module--navItemGroup--CtPAh"><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://www.confluent.io/blog/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Blog</h4><p class="style-module--subline--cWWdf">The Confluent blog</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/learn-more/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys"><span class="style-module--indicator--WTN0y">NEW</span>Learn More</h4><p class="style-module--subline--cWWdf">Other ways to learn</p></div></div></a><span/></li></ul></li></ul></div></div></div><div class="style-module--accordionItemWrapper--Ds610 style-module--collapsed--ydSG7 style-module--mobileAccordionItem--m9OUc"><div class="style-module--accordionItem--Bex3L style-module--accordionItem--dbe+-"><div class="style-module--accordionTitle--gPya- style-module--titleClassName--CWu6r">Build<div class="IconArrow-module--arrow--ULhik IconArrow-module--down--49+OW IconArrow-module--md--9BJoY IconArrow-module--theme_white--kOiXH style-module--indicator--FKh5h"/></div><div class="style-module--accordionPanel--NujUc"><ul class="style-module--menuGroup--rr-S+"><li class="style-module--navItemHeaderContainer--vxHRS"><div class="style-module--navItemGroupWrapper--utjJn style-module--roundTopBorderForTablets--SlQz4"><div><div class="style-module--titleWithOutMarginTop--QErYg">Design. Build. Run.</div><span class="style-module--subline--cWWdf">Build a client app, explore use cases, and build on our demos and resources</span><a class="style-module--navItemGroupCta--YzEUp" data-test-id="link" href="https://developer.confluent.io/#designbuildrun">Start Building<span/></a></div></div></li><li class="style-module--navItemLinearGroupContainer--E8b-c"><ul class="style-module--navItemGroup--CtPAh"><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://developer.confluent.io/get-started/java/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Language Guides</h4><p class="style-module--subline--cWWdf">Build apps in your favorite language</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://developer.confluent.io/tutorials/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Tutorials</h4><p class="style-module--subline--cWWdf">Hands-on stream processing examples</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://developer.confluent.io/demos-examples/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Demos</h4><p class="style-module--subline--cWWdf">More resources to get you started</p></div></div></a><span/></li></ul></li><li class="style-module--navItemNonLinearGroupContainer--0LVfk style-module--roundBottomBorderForTablets--bEt6C"><ul class="style-module--navItemGroup--CtPAh"><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://developer.confluent.io/get-started/java/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Language Guides</h4><p class="style-module--subline--cWWdf">Build apps in your favorite language</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://developer.confluent.io/tutorials/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Tutorials</h4><p class="style-module--subline--cWWdf">Hands-on stream processing examples</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://developer.confluent.io/demos-examples/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Demos</h4><p class="style-module--subline--cWWdf">More resources to get you started</p></div></div></a><span/></li></ul></li></ul></div></div></div><div class="style-module--accordionItemWrapper--Ds610 style-module--collapsed--ydSG7 style-module--mobileAccordionItem--m9OUc"><div class="style-module--accordionItem--Bex3L style-module--accordionItem--dbe+-"><div class="style-module--accordionTitle--gPya- style-module--titleClassName--CWu6r">Community<div class="IconArrow-module--arrow--ULhik IconArrow-module--down--49+OW IconArrow-module--md--9BJoY IconArrow-module--theme_white--kOiXH style-module--indicator--FKh5h"/></div><div class="style-module--accordionPanel--NujUc"><ul class="style-module--menuGroup--rr-S+"><li class="style-module--navItemHeaderContainer--vxHRS"><div class="style-module--navItemGroupWrapper--utjJn style-module--roundTopBorderForTablets--SlQz4"><div><div class="style-module--titleWithOutMarginTop--QErYg">Join the Community</div><span class="style-module--subline--cWWdf">Confluent proudly supports the global community of streaming platforms, real-time data streams, Apache Kafka®️, and its ecosystems</span><a class="style-module--navItemGroupCta--YzEUp" data-test-id="link" href="/community/">Learn More<span/></a></div></div></li><li class="style-module--navItemLinearGroupContainer--E8b-c"><ul class="style-module--navItemGroup--CtPAh"><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" rel=" noreferrer" href="https://www.meetup.com/pro/confluent/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Meetups &amp; Events</h4><p class="style-module--subline--cWWdf">Kafka and data streaming community</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/community/ask-the-community/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Ask the Community</h4><p class="style-module--subline--cWWdf">Community forums and Slack channels</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/catalysts/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Community Catalysts</h4><p class="style-module--subline--cWWdf">Sharing expertise with the community</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/case-studies/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys"><span class="style-module--indicator--WTN0y">NEW</span>Community Use Cases</h4><p class="style-module--subline--cWWdf">Real-world Examples of Apache Kafka® and Flink® in Action</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/newsletter/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">DevX Newsletter</h4><p class="style-module--subline--cWWdf">Bi-weekly newsletter with Apache Kafka® resources, news from the community, and fun links. </p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" rel=" noreferrer" href="https://www.datastreamingawards.io/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Data Streaming Awards</h4><p class="style-module--subline--cWWdf">Nominate amazing use cases and view previous winners</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://www.confluent.io/events/current/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys"><span class="style-module--indicator--WTN0y">NEW</span>Current 2024</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" rel=" noreferrer" href="https://www.kafka-summit.org/events/kafka-summit-bangalore-2024/register"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys"><span class="style-module--indicator--WTN0y">NEW</span>Kafka Summit 2024 - Bangalore</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://www.confluent.io/events/kafka-summit-london-2024/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys"><span class="style-module--indicator--WTN0y">NEW</span>Kafka Summit 2024 - London</h4><p class="style-module--subline--cWWdf">View sessions and slides</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://www.confluent.io/events/current/2023/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Current 2023</h4><p class="style-module--subline--cWWdf">View sessions and slides</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" rel=" noreferrer" href="https://www.kafka-summit.org/events/kafka-summit-london-2023/about"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Kafka Summit 2023</h4><p class="style-module--subline--cWWdf">View sessions and slides</p></div></div></a><span/></li></ul></li><li class="style-module--navItemNonLinearGroupContainer--0LVfk style-module--roundBottomBorderForTablets--bEt6C"><ul class="style-module--navItemGroup--CtPAh"><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" rel=" noreferrer" href="https://www.meetup.com/pro/confluent/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Meetups &amp; Events</h4><p class="style-module--subline--cWWdf">Kafka and data streaming community</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/community/ask-the-community/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Ask the Community</h4><p class="style-module--subline--cWWdf">Community forums and Slack channels</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/catalysts/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Community Catalysts</h4><p class="style-module--subline--cWWdf">Sharing expertise with the community</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/case-studies/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys"><span class="style-module--indicator--WTN0y">NEW</span>Community Use Cases</h4><p class="style-module--subline--cWWdf">Real-world Examples of Apache Kafka® and Flink® in Action</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/newsletter/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">DevX Newsletter</h4><p class="style-module--subline--cWWdf">Bi-weekly newsletter with Apache Kafka® resources, news from the community, and fun links. </p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" rel=" noreferrer" href="https://www.datastreamingawards.io/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Data Streaming Awards</h4><p class="style-module--subline--cWWdf">Nominate amazing use cases and view previous winners</p></div></div></a><span/></li></ul><ul class="style-module--navItemGroup--CtPAh"><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://www.confluent.io/events/current/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys"><span class="style-module--indicator--WTN0y">NEW</span>Current 2024</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" rel=" noreferrer" href="https://www.kafka-summit.org/events/kafka-summit-bangalore-2024/register"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys"><span class="style-module--indicator--WTN0y">NEW</span>Kafka Summit 2024 - Bangalore</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://www.confluent.io/events/kafka-summit-london-2024/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys"><span class="style-module--indicator--WTN0y">NEW</span>Kafka Summit 2024 - London</h4><p class="style-module--subline--cWWdf">View sessions and slides</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://www.confluent.io/events/current/2023/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Current 2023</h4><p class="style-module--subline--cWWdf">View sessions and slides</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" rel=" noreferrer" href="https://www.kafka-summit.org/events/kafka-summit-london-2023/about"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Kafka Summit 2023</h4><p class="style-module--subline--cWWdf">View sessions and slides</p></div></div></a><span/></li></ul></li></ul></div></div></div><a class="style-module--linkButton--PEIQe" data-test-id="link" rel="noopener" target="_blank" href="https://docs.confluent.io/home/overview.html">Docs</a><a class="style-module--linkButton--PEIQe style-module--search--TuMK-" data-test-id="link" href="/search/">Search</a></nav></div><nav class="style-module--nav--9nLqL" data-swiftype-index="false"><ul class="style-module--unorderedList--lbeVr"><li class="style-module--navGroup--J6HXn"><a data-test-id="link" href="https://developer.confluent.io/courses">Courses</a><ul class="style-module--menuGroup--rr-S+"><li><div class="style-module--navItemGroupWrapper--utjJn style-module--roundBottomLeftBorder--1K2n3"><div><div class="style-module--titleWithOutMarginTop--QErYg">What are the courses?</div><span class="style-module--subline--cWWdf">Video courses covering Apache Kafka basics, advanced concepts, setup and use cases, and everything in between.</span><a class="style-module--navItemGroupCta--YzEUp" data-test-id="link" href="/courses/">Learning Pathways (24)<span/></a></div></div></li><li><ul class="style-module--navItemGroup--CtPAh"><li class="style-module--titleWithOutMarginTop--QErYg">New Courses</li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/courses/microservices/from-monoliths-to-microservices/"><div class="style-module--navItem--1kjpc"><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys"><span class="style-module--indicator--WTN0y">NEW</span>Designing Event-Driven Microservices</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/courses/apache-flink/intro/"><div class="style-module--navItem--1kjpc"><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys"><span class="style-module--indicator--WTN0y">NEW</span>Apache Flink® 101</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/courses/flink-java/overview/"><div class="style-module--navItem--1kjpc"><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys"><span class="style-module--indicator--WTN0y">NEW</span>Building Flink® Apps in Java</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/courses/hybrid-cloud/intro/"><div class="style-module--navItem--1kjpc"><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys"><span class="style-module--indicator--WTN0y">NEW</span>Hybrid and Multicloud Architecture</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/courses/data-streaming-systems/overview/"><div class="style-module--navItem--1kjpc"><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys"><span class="style-module--indicator--WTN0y">NEW</span>Mastering Production Data Streaming Systems with Apache Kafka®</h4></div></div></a><span/></li></ul></li><li><ul class="style-module--navItemGroup--CtPAh"><li class="style-module--titleWithOutMarginTop--QErYg">Featured Courses</li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://developer.confluent.io/courses/apache-kafka/events/"><div class="style-module--navItem--1kjpc"><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Kafka® 101</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://developer.confluent.io/courses/kafka-connect/intro/"><div class="style-module--navItem--1kjpc"><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Kafka® Connect 101</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://developer.confluent.io/courses/kafka-streams/get-started/"><div class="style-module--navItem--1kjpc"><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Kafka Streams 101</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://developer.confluent.io/courses/schema-registry/key-concepts"><div class="style-module--navItem--1kjpc"><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Schema Registry 101</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://developer.confluent.io/courses/ksqldb/intro/"><div class="style-module--navItem--1kjpc"><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">ksqlDB 101</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://developer.confluent.io/courses/data-mesh/intro/"><div class="style-module--navItem--1kjpc"><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Data Mesh 101</h4></div></div></a><span/></li></ul></li></ul></li><li class="style-module--navGroup--J6HXn"><a data-test-id="link" href="https://developer.confluent.io/learn/">Learn</a><ul class="style-module--menuGroup--rr-S+"><li><div class="style-module--navItemGroupWrapper--utjJn style-module--roundBottomLeftBorder--1K2n3"><div><div class="style-module--titleWithOutMarginTop--QErYg">Pick your learning path</div><span class="style-module--subline--cWWdf">A wide range of resources to get you started</span><a class="style-module--navItemGroupCta--YzEUp" data-test-id="link" href="https://developer.confluent.io/courses">Start Learning<span/></a></div></div></li><li><ul class="style-module--navItemGroup--CtPAh"><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/learn/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Articles</h4><p class="style-module--subline--cWWdf">Deep-Dives into key concepts</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/patterns/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Patterns</h4><p class="style-module--subline--cWWdf">Architectures for event streaming</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/faq/apache-kafka/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">FAQs</h4><p class="style-module--subline--cWWdf">Q &amp; A about Kafka® and its ecosystem</p></div></div></a><span/></li></ul></li><li><ul class="style-module--navItemGroup--CtPAh"><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://www.confluent.io/blog/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Blog</h4><p class="style-module--subline--cWWdf">The Confluent blog</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/learn-more/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys"><span class="style-module--indicator--WTN0y">NEW</span>Learn More</h4><p class="style-module--subline--cWWdf">Other ways to learn</p></div></div></a><span/></li></ul></li></ul></li><li class="style-module--navGroup--J6HXn"><a data-test-id="link" href="https://developer.confluent.io/#designbuildrun">Build</a><ul class="style-module--menuGroup--rr-S+"><li><div class="style-module--navItemGroupWrapper--utjJn style-module--roundBottomLeftBorder--1K2n3"><div><div class="style-module--titleWithOutMarginTop--QErYg">Design. Build. Run.</div><span class="style-module--subline--cWWdf">Build a client app, explore use cases, and build on our demos and resources</span><a class="style-module--navItemGroupCta--YzEUp" data-test-id="link" href="https://developer.confluent.io/#designbuildrun">Start Building<span/></a></div></div></li><li><ul class="style-module--navItemGroup--CtPAh"><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://developer.confluent.io/get-started/java/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Language Guides</h4><p class="style-module--subline--cWWdf">Build apps in your favorite language</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://developer.confluent.io/tutorials/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Tutorials</h4><p class="style-module--subline--cWWdf">Hands-on stream processing examples</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://developer.confluent.io/demos-examples/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Demos</h4><p class="style-module--subline--cWWdf">More resources to get you started</p></div></div></a><span/></li></ul></li></ul></li><li class="style-module--navGroup--J6HXn"><a data-test-id="link" href="/community/">Community</a><ul class="style-module--menuGroup--rr-S+"><li><div class="style-module--navItemGroupWrapper--utjJn style-module--roundBottomLeftBorder--1K2n3"><div><div class="style-module--titleWithOutMarginTop--QErYg">Join the Community</div><span class="style-module--subline--cWWdf">Confluent proudly supports the global community of streaming platforms, real-time data streams, Apache Kafka®️, and its ecosystems</span><a class="style-module--navItemGroupCta--YzEUp" data-test-id="link" href="/community/">Learn More<span/></a></div></div></li><li><ul class="style-module--navItemGroup--CtPAh"><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" rel=" noreferrer" href="https://www.meetup.com/pro/confluent/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Meetups &amp; Events</h4><p class="style-module--subline--cWWdf">Kafka and data streaming community</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/community/ask-the-community/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Ask the Community</h4><p class="style-module--subline--cWWdf">Community forums and Slack channels</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/catalysts/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Community Catalysts</h4><p class="style-module--subline--cWWdf">Sharing expertise with the community</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/case-studies/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys"><span class="style-module--indicator--WTN0y">NEW</span>Community Use Cases</h4><p class="style-module--subline--cWWdf">Real-world Examples of Apache Kafka® and Flink® in Action</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="/newsletter/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">DevX Newsletter</h4><p class="style-module--subline--cWWdf">Bi-weekly newsletter with Apache Kafka® resources, news from the community, and fun links. </p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" rel=" noreferrer" href="https://www.datastreamingawards.io/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Data Streaming Awards</h4><p class="style-module--subline--cWWdf">Nominate amazing use cases and view previous winners</p></div></div></a><span/></li></ul></li><li><ul class="style-module--navItemGroup--CtPAh"><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://www.confluent.io/events/current/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys"><span class="style-module--indicator--WTN0y">NEW</span>Current 2024</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" rel=" noreferrer" href="https://www.kafka-summit.org/events/kafka-summit-bangalore-2024/register"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys"><span class="style-module--indicator--WTN0y">NEW</span>Kafka Summit 2024 - Bangalore</h4></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://www.confluent.io/events/kafka-summit-london-2024/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys"><span class="style-module--indicator--WTN0y">NEW</span>Kafka Summit 2024 - London</h4><p class="style-module--subline--cWWdf">View sessions and slides</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" href="https://www.confluent.io/events/current/2023/"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Current 2023</h4><p class="style-module--subline--cWWdf">View sessions and slides</p></div></div></a><span/></li><li class="style-module--navItemContainer--tHsKP"><a class="style-module--navItemLink--czujm" data-test-id="link" rel=" noreferrer" href="https://www.kafka-summit.org/events/kafka-summit-london-2023/about"><div class="style-module--navItem--1kjpc"><span class=""/><div class="style-module--navItemLinkSection--GP6lc"><h4 class="style-module--title--acZys">Kafka Summit 2023</h4><p class="style-module--subline--cWWdf">View sessions and slides</p></div></div></a><span/></li></ul></li></ul></li><li class="style-module--navGroup--J6HXn"><a data-test-id="link" href="https://docs.confluent.io/home/overview.html">Docs</a></li><a class="style-module--linkButton--PEIQe style-module--search--TuMK-" data-test-id="link" href="/search/"><span class="style-module--visuallyHidden--PY0rE">Search</span></a><li class="style-module--quickstart--8VLFU"><a role="button" class="Button-module--button--A-Z1p Button-module--whiteOutline--pRCyA" data-test-id="link" href="https://www.confluent.io/get-started/">Get Started Free</a></li></ul></nav></div></header><div class="style-module--page--tdXas"><div style="padding-top:81px"><div class="style-module--container--Sgq1e"><div class="style-module--main--IpSMy"><div class="style-module--video--J1VNs"/><div class="style-module--content--pAlfP"><div class=""><div class="style-module--back--qu93+"><a data-test-id="link" href="/courses/">‹ Back to <span>courses</span></a><button type="button" class="Button-module--button--A-Z1p Button-module--oceanOutline--vrbqo">View Transcript</button></div><div class="style-module--titleContainer--RAIck"><a class="style-module--tag--iQNTY" data-test-id="link" href="/courses/kafka-connect/intro/">course: Kafka Connect 101</a><div class="style-module--title--TyoOk"><div><h1 data-swiftype-name="title" data-swiftype-type="string">Connectors, Configuration, Converters, and Transforms</h1><div class="style-module--videoLength--19Kph style-module--videoLength--axWtq">9 min</div></div><div class="style-module--titleSocial--nBB2z"><button aria-label="twitter" class="react-share__ShareButton IconShare-module--button--0awKi" style="background-color:transparent;border:none;padding:0;font:inherit;color:inherit;cursor:pointer"><svg viewbox="0 0 64 64" width="36" height="36"><circle cx="32" cy="32" r="31" fill="#00aced"/><path d="M48,22.1c-1.2,0.5-2.4,0.9-3.8,1c1.4-0.8,2.4-2.1,2.9-3.6c-1.3,0.8-2.7,1.3-4.2,1.6 C41.7,19.8,40,19,38.2,19c-3.6,0-6.6,2.9-6.6,6.6c0,0.5,0.1,1,0.2,1.5c-5.5-0.3-10.3-2.9-13.5-6.9c-0.6,1-0.9,2.1-0.9,3.3 c0,2.3,1.2,4.3,2.9,5.5c-1.1,0-2.1-0.3-3-0.8c0,0,0,0.1,0,0.1c0,3.2,2.3,5.8,5.3,6.4c-0.6,0.1-1.1,0.2-1.7,0.2c-0.4,0-0.8,0-1.2-0.1 c0.8,2.6,3.3,4.5,6.1,4.6c-2.2,1.8-5.1,2.8-8.2,2.8c-0.5,0-1.1,0-1.6-0.1c2.9,1.9,6.4,2.9,10.1,2.9c12.1,0,18.7-10,18.7-18.7 c0-0.3,0-0.6,0-0.8C46,24.5,47.1,23.4,48,22.1z" fill="white"/></svg></button><button aria-label="linkedin" class="react-share__ShareButton IconShare-module--button--0awKi" style="background-color:transparent;border:none;padding:0;font:inherit;color:inherit;cursor:pointer"><svg viewbox="0 0 64 64" width="36" height="36"><circle cx="32" cy="32" r="31" fill="#007fb1"/><path d="M20.4,44h5.4V26.6h-5.4V44z M23.1,18c-1.7,0-3.1,1.4-3.1,3.1c0,1.7,1.4,3.1,3.1,3.1 c1.7,0,3.1-1.4,3.1-3.1C26.2,19.4,24.8,18,23.1,18z M39.5,26.2c-2.6,0-4.4,1.4-5.1,2.8h-0.1v-2.4h-5.2V44h5.4v-8.6 c0-2.3,0.4-4.5,3.2-4.5c2.8,0,2.8,2.6,2.8,4.6V44H46v-9.5C46,29.8,45,26.2,39.5,26.2z" fill="white"/></svg></button><button aria-label="facebook" class="react-share__ShareButton IconShare-module--button--0awKi" style="background-color:transparent;border:none;padding:0;font:inherit;color:inherit;cursor:pointer"><svg viewbox="0 0 64 64" width="36" height="36"><circle cx="32" cy="32" r="31" fill="#3b5998"/><path d="M34.1,47V33.3h4.6l0.7-5.3h-5.3v-3.4c0-1.5,0.4-2.6,2.6-2.6l2.8,0v-4.8c-0.5-0.1-2.2-0.2-4.1-0.2 c-4.1,0-6.9,2.5-6.9,7V28H24v5.3h4.6V47H34.1z" fill="white"/></svg></button></div></div></div></div><div class="style-module--presenterContainer--kCvNU"><div class="style-module--presenter--yzfWg"><div data-gatsby-image-wrapper="" class="gatsby-image-wrapper gatsby-image-wrapper-constrained"><div style="max-width:128px;display:block"><img alt="" role="presentation" aria-hidden="true" src="data:image/svg+xml;charset=utf-8,%3Csvg height='128' width='128' xmlns='http://www.w3.org/2000/svg' version='1.1'%3E%3C/svg%3E" style="max-width:100%;display:block;position:static"/></div><img aria-hidden="true" data-placeholder-image="" style="opacity:1;transition:opacity 500ms linear;background-color:transparent;top:0;left:0;bottom:0;right:0" decoding="async" src="data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEASABIAAD/2wBDAAoHBwgHBgoICAgLCgoLDhgQDg0NDh0VFhEYIx8lJCIfIiEmKzcvJik0KSEiMEExNDk7Pj4+JS5ESUM8SDc9Pjv/2wBDAQoLCw4NDhwQEBw7KCIoOzs7Ozs7Ozs7Ozs7Ozs7Ozs7Ozs7Ozs7Ozs7Ozs7Ozs7Ozs7Ozs7Ozs7Ozs7Ozs7Ozv/wAARCAAUABQDASIAAhEBAxEB/8QAGgABAAIDAQAAAAAAAAAAAAAAAAQGAgMFB//EACYQAAIBBAIABQUAAAAAAAAAAAECAwAEBRIRIQYiMUFxE1FhgZH/xAAXAQADAQAAAAAAAAAAAAAAAAACAwQG/8QAHhEBAAICAQUAAAAAAAAAAAAAAQACAxExBCFRYbH/2gAMAwEAAhEDEQA/AMb76IyAix8qKbkRwxyv5tNtmZiD9lXr5rRNi4vDWTgvGyC3tuWVLuOVQTqxA2/RINRsrFNjM3YXkrCa1hO026g+UDX09+ia7ufMN7jWtVAkW4K9Qr3r1+Bx6VNa7Ww1lWKhajvmTHscQ0jlcch5YkkIOz/aVTZ/F+StZTCk0ciL0pZAeuaU9iK4zUjT3dxLJGskpcPGA23fPPrV18Tym2x0ywgJpa8qVHBHVKUzAD1qPl+ws/bCa9Tyz2Hv80pStg46LwSDbP/Z" alt=""/><picture><source type="image/webp" data-srcset="https://images.ctfassets.net/gt6dp23g0g38/5xPlcrlqRVHoIw3RVIeRIB/6bc2ee9336eb98204f781b133a62ad1b/danica-fine.jpg?w=128&amp;h=128&amp;q=70&amp;fm=webp&amp;bg=transparent 128w" sizes="(min-width: 128px) 128px, 100vw"/><img data-gatsby-image-ssr="" data-main-image="" style="opacity:0" sizes="(min-width: 128px) 128px, 100vw" decoding="async" loading="lazy" data-src="https://images.ctfassets.net/gt6dp23g0g38/5xPlcrlqRVHoIw3RVIeRIB/6bc2ee9336eb98204f781b133a62ad1b/danica-fine.jpg?w=128&amp;h=128&amp;fl=progressive&amp;q=70&amp;fm=jpg&amp;bg=transparent" data-srcset="https://images.ctfassets.net/gt6dp23g0g38/5xPlcrlqRVHoIw3RVIeRIB/6bc2ee9336eb98204f781b133a62ad1b/danica-fine.jpg?w=128&amp;h=128&amp;fl=progressive&amp;q=70&amp;fm=jpg&amp;bg=transparent 128w" alt="Danica Fine"/></picture><noscript><picture><source type="image/webp" srcset="https://images.ctfassets.net/gt6dp23g0g38/5xPlcrlqRVHoIw3RVIeRIB/6bc2ee9336eb98204f781b133a62ad1b/danica-fine.jpg?w=128&amp;h=128&amp;q=70&amp;fm=webp&amp;bg=transparent 128w" sizes="(min-width: 128px) 128px, 100vw"/><img data-gatsby-image-ssr="" data-main-image="" style="opacity:0" sizes="(min-width: 128px) 128px, 100vw" decoding="async" loading="lazy" src="https://images.ctfassets.net/gt6dp23g0g38/5xPlcrlqRVHoIw3RVIeRIB/6bc2ee9336eb98204f781b133a62ad1b/danica-fine.jpg?w=128&amp;h=128&amp;fl=progressive&amp;q=70&amp;fm=jpg&amp;bg=transparent" srcset="https://images.ctfassets.net/gt6dp23g0g38/5xPlcrlqRVHoIw3RVIeRIB/6bc2ee9336eb98204f781b133a62ad1b/danica-fine.jpg?w=128&amp;h=128&amp;fl=progressive&amp;q=70&amp;fm=jpg&amp;bg=transparent 128w" alt="Danica Fine"/></picture></noscript></div><header class="style-module--info--4JN+9"><h4 class="style-module--name--mhzT6">Danica Fine</h4><p class="style-module--role--q5BaI">Senior Developer Advocate (Presenter)</p></header></div></div><div data-swiftype-name="body" data-swiftype-type="text" class="style-module--mdHtml--ZuBR8 style-module--md--rjKdq"><h2 id="inside-kafka-connect">Inside Kafka Connect<a aria-hidden="true" tabindex="-1" href="#inside-kafka-connect"><span class="header-link"/></a></h2> <p>Kafka Connect is built around a pluggable architecture of several components, which together provide very flexible integration pipelines. To get the most out of Kafka Connect it’s important to understand these components and their roles:</p> <li>Connectors are responsible for the interaction between Kafka Connect and the external technology it’s being integrated with</li> <li>Converters handle the serialization and deserialization of data</li> <li>Transformations can optionally apply one or more transformations to the data passing through the pipeline</li> <h2 id="connectors">Connectors<a aria-hidden="true" tabindex="-1" href="#connectors"><span class="header-link"/></a></h2> <p>The key component of any Kafka Connect pipeline is a connector instance which is a logical job that defines where data should be copied to and from. All of the classes that implement or are used by a connector instance are defined in its connector plugin. Written by the community, a vendor, or occasionally written bespoke by the user, the plugin integrates Kafka Connect with a particular technology. These plugins are reusable components that you can download, install, and use without writing code.</p> <p>For example:</p> <li>The Debezium MySQL source connector uses the MySQL bin log to read events from the database and stream these to Kafka Connect</li> <li>The Elasticsearch sink connector takes data from Kafka Connect, and using the Elasticsearch APIs, writes the data to Elasticsearch</li> <li>The S3 connector from Confluent can act as both a source and sink connector, writing data to S3 or reading it back in</li> <p>A SOURCE connector plugin knows how to talk to a specific SOURCE data system and generate records that Kafka Connect then writes into Kafka. On the downstream side, the connector instance configuration specifies the topics to be consumed and Kafka Connect reads those topics and sends them to the SINK connector that knows how to send those records to a specific SINK data system.</p> <p>So the connectors know how to work with the records and talk to the external data system, but Kafka Connect workers act as the conductor and take care of the rest. We will define what a worker is shortly.</p> <h2 id="add-a-connector-instance-with-the-rest-api">Add a Connector Instance with the REST API<a aria-hidden="true" tabindex="-1" href="#add-a-connector-instance-with-the-rest-api"><span class="header-link"/></a></h2> <p>To specify a connector you include its name in your configuration—each connector's documentation will give you the particular classname string to use. As you may expect, connectors have different configuration properties specific to the technology with which they’re integrating. A cloud connector will need to know the region, the credentials, and the endpoint to use. A database connector will need to know the names of the tables, the database hostname, and so on.</p> <p>Here’s an example that creates an Elasticsearch sink connector instance with a call to Kafka Connect’s REST API:</p> <div class="gatsby-highlight" data-language="sh"><pre class="language-sh"><span class="language-sh"><span class="token function">curl</span> <span class="token parameter variable">-X</span> PUT <span class="token parameter variable">-H</span> <span class="token string">"Content-Type:application/json"</span> http://localhost:8083/connectors/sink-elastic-01/config <span class="token punctuation">\</span> <span class="token parameter variable">-d</span> <span class="token string">'{ "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector", "topics" : "orders", "connection.url" : "http://elasticsearch:9200", "type.name" : "_doc", "key.ignore" : "false", "schema.ignore" : "true" <h2 id="add-a-connector-instance-with-ksqldb">Add a Connector Instance with ksqlDB<a aria-hidden="true" tabindex="-1" href="#add-a-connector-instance-with-ksqldb"><span class="header-link"/></a></h2> <p>You can also use ksqlDB to manage connectors.</p> <p>Here is the syntax for adding the previous Elasticsearch sink connector instance:</p> <div class="gatsby-highlight" data-language="sh"><pre class="language-sh"><span class="language-sh">CREATE SINK CONNECTOR sink-elastic-01 WITH <span class="token punctuation">(</span> <span class="token string">'connector.class'</span> <span class="token operator">=</span> <span class="token string">'io.confluent.connect.elasticsearch.ElasticsearchSinkConnector'</span>, <span class="token string">'topics'</span> <span class="token operator">=</span> <span class="token string">'orders'</span>, <span class="token string">'connection.url'</span> <span class="token operator">=</span> <span class="token string">'http://elasticsearch:9200'</span>, <span class="token string">'type.name'</span> <span class="token operator">=</span> <span class="token string">'_doc'</span>, <span class="token string">'key.ignore'</span> <span class="token operator">=</span> <span class="token string">'false'</span>, <span class="token string">'schema.ignore'</span> <span class="token operator">=</span> <span class="token string">'true'</span> <h2 id="add-a-connector-instance-with-the-console-ui">Add a Connector Instance with the Console UI<a aria-hidden="true" tabindex="-1" href="#add-a-connector-instance-with-the-console-ui"><span class="header-link"/></a></h2> <p>In addition to using the Kafka Connect REST API directly, you can add connector instances using the Confluent Cloud console.</p> <h2 id="what-is-the-role-of-the-connector">What Is the Role of the Connector?<a aria-hidden="true" tabindex="-1" href="#what-is-the-role-of-the-connector"><span class="header-link"/></a></h2> <p>It’s important to understand that the connector plugins themselves don't read from or write to (consume/produce) Kafka itself. The plugins just provide the interface between Kafka and the external technology. This is a deliberate design.</p> <li>Source connectors interface with the source API and extract the payload + schema of the data, and pass this internally as a generic representation of the data.</li> <li>Sink connectors work in reverse—they take a generic representation of the data, and the sink connector plugin writes that to the target system using its API.</li> <p>Kafka Connect and its underlying components take care of writing data received from source connectors to Kafka topics as well as reading data from Kafka topics and passing it to sink connectors.</p> <p>Now, this is all hidden from the user—when you add a new connector instance, that’s all you need to configure and Kafka Connect does the rest to get the data flowing. Converters are the next piece of the puzzle and it is important to understand them to help you avoid common pitfalls with Kafka Connect. Technically, transforms sit between connectors and converters, but we’ll visit those later.</p> <h2 id="converters-serializedeserialize-the-data">Converters Serialize/Deserialize the Data<a aria-hidden="true" tabindex="-1" href="#converters-serializedeserialize-the-data"><span class="header-link"/></a></h2> <p>Converters are responsible for the serialization and deserialization of data flowing between Kafka Connect and Kafka itself. You’ll sometimes see similar components referred to as SerDes (“SerializerDeserializer”) in Kafka Streams, or just plain old serializers and deserializers in the Kafka Client libraries.</p> <p>There are a ton of different converters available, but some common ones include:</p> <li>Avro – io.confluent.connect.avro.AvroConverter</li> <li>Protobuf – io.confluent.connect.protobuf.ProtobufConverter</li> <li>String – org.apache.kafka.connect.storage.StringConverter</li> <li>JSON – org.apache.kafka.connect.json.JsonConverter</li> <li>JSON Schema – io.confluent.connect.json.JsonSchemaConverter</li> <li>ByteArray – org.apache.kafka.connect.converters.ByteArrayConverter</li> <p>While Kafka doesn’t care about how you serialize your data (as far as it’s concerned, it’s just a series of bytes), you should care about how you serialize your data! In the same way that you would take a carefully considered approach to how you design your services and model your data, you should also be deliberate in your serialization approach.</p> <h2 id="serialization-and-schemas">Serialization and Schemas<a aria-hidden="true" tabindex="-1" href="#serialization-and-schemas"><span class="header-link"/></a></h2> <p>As well as managing the straightforward matter of serializing data flowing into Kafka and deserializing it on its way out, converters have a crucial role to play in the persistence of schemas. Almost all data that we deal with has a schema; it’s up to us whether we choose to acknowledge that in our designs or not. You can consider schemas as the API between applications and components of a pipeline. Schemas are the contract between one component in the pipeline and another, describing the shape and form of the data.</p> <p>When you ingest data from a source such as a database, as well as the rows of data, you have the metadata that describes the fields—the data types, their names, etc. Having this schema metadata is valuable, and you will want to retain it in an efficient manner. A great way to do this is by using a serialization method such as Avro, Protobuf, or JSON Schema. All three of these will serialize the data onto a Kafka topic and then store the schema separately in the Confluent Schema Registry. By storing the schema for data, you can easily utilize it in your consuming applications and pipelines. You can also use it to enforce data hygiene in the pipeline by ensuring that only data that is compatible with the schema is stored on a given topic.</p> <p>You can opt to use serialization formats that don’t store schemas like JSON, string, and byte array, and in some cases, these are valid. If you use these, just make sure that you are doing so for deliberate reasons and have considered how else you will handle schema information.</p> <h2 id="converters-specified-for-key-and-value">Converters Specified for Key and Value<a aria-hidden="true" tabindex="-1" href="#converters-specified-for-key-and-value"><span class="header-link"/></a></h2> <p>Converters are specified separately for the value of a message, and its key:</p> <div class="gatsby-highlight" data-language="sh"><pre class="language-sh"><span class="language-sh"><span class="token assign-left variable">key.converter</span><span class="token operator">=</span>org.apache.kafka.connect.storage.StringConverter <span class="token assign-left variable">value.converter</span><span class="token operator">=</span>org.apache.kafka.connect.storage.StringConverter <span class="token assign-left variable">value.converter.schema.registry.url</span><span class="token operator">=</span>http://localhost:8081</span></pre></div> <p>Note that these converters are set as a global default per Connect worker, but they can be overridden per connector instance.</p> <h2 id="single-message-transforms">Single Message Transforms<a aria-hidden="true" tabindex="-1" href="#single-message-transforms"><span class="header-link"/></a></h2> <p>The third and final key component in Kafka Connect is the transform piece. Unlike connectors and converters, these are entirely optional. You can use them to modify data from a source connector before it is written to Kafka, and modify data read from Kafka before it’s written to the sink. Transforms operate over individual messages as they move, so they’re known as Single Message Transforms or SMTs.</p> <p>Common uses for SMTs include:</p> <li>Dropping fields from data at ingest, such as personally identifiable information (PII) if specified by the system requirements</li> <li>Adding metadata information such as lineage to data ingested through Kafka Connect</li> <li>Changing field data types</li> <li>Modifying the topic name to include a timestamp</li> <li>Renaming fields</li>