Apache Flink

Customers are becoming digital, and digital needs data NOW. Fast forward, data analytics quickly evolved from legacy Data Warehouses to Big Data, but always looking at "what happened", a retrospective view. Businesses need fresh data to be faster and make smarter decisions, and that means real-time data. I am happy to introduce "Stream Processing Landscape", a general guideline to understand how Apache Flink, a leading open source engine for real-time data, is setting the pace for data streaming processing and fits into the enterprise ecosystem. 🟢 Structured: well-know and well-governed data source, structured data is pointed to be around 20% of all the corporate data in the world. It is mainly stored in databases, with defined schemas. Apache Flink commonly leverages a CDC strategy to consume data in real-time from databases. 🟢 Unstructured: the remaining 80% of the enterprise data is a combination of unstructured data formats, in many ways. Cost-effective and scalable Big Data solutions, such as cloud-native storage and Apache Kafka, helps companies to safe guard and store years of logs, machine data and images. 🟢 Enterprise Apps: streaming data should integrate with the application ecosystem, triggering actions for best next offers, ad-hoc advertisement and up-selling opportunities. Specialized solutions for marketing, point of sales and enterprise management can be augmented with real-time data and AI. 🟢 Data Ecosystem: Apache Flink leverages most the robust and mature frameworks and engines currently available in the data ecosystem, including open data formats like Apache Parquet, and new Big Data management approaches such as Apache Iceberg and Fluss for Lakehouses. These open standards ensure interoperability and freedom of choice to adopt any tool, any vendor. #StreamProcessing #RealtimeData #ApacheFlink #ApacheIceberg #Lakehouse

This article sat in my Medium drafts for more than 1 month—I almost forgot to hit publish! I’m excited (and a little nervous) to finally share my first deep-dive into real-time data pipeline — and also my first time using Kafka, Debezium, Flink, Scala, and ClickHouse all together. The goal? Keep today’s data blazing-fast while storing history cheaply. In the write-up, I walk through: 1. Setting up CDC from PostgreSQL to Kafka. 2. Streaming writes into ClickHouse with Apache Flink Scala job in Hot Layer. 3. Splitting Local vs. Remote storage inside ClickHouse. 4. Daily Apache Airflow jobs to run task in Cold Layer: Bronze Hot -> Bronze Cold -> Silver Cold -> Gold Cold. 5. A simple dashboard to bring it all to life. It’s far from perfect, but I learned a ton—and I hope this write-up saves you a few rabbit holes (and headaches) if you’re building something similar. #Kafka #DataEngineering #RealtimeAnalysis #Clickhouse #ApacheFlink

Hallo Nederland! 🇳🇱 Amsterdam is ready to receive some "real-time" data with Apache Flink® Lab Day hosted by Ververica 🚀 I am excited to bring to life the very first Lab Day powered by Apache Flink, and the beautiful city of Amsterdam will be our first stop 🌏 🔥 What to expect from Lab Day hosted by Ververica | Original creators of Apache Flink®? ⏩ Leverage Apache Flink® to develop an end-to-end use case. ⏩ Build your first streaming application. ⏩ Connect to most-common data source such as Kafka and databases. ⏩ Become a Streaming Champion. 📅 Thursday, September 18, 2025 🕣 8:30 AM - 12:00 PM 📍 Central Amsterdam, Netherlands Secure your spot here: https://lnkd.in/dec3SgQ3 Bouke van der Meer Francie Kastl Jaime López Maciej Mojsiewicz Michael R Misurell Spencer Arnold Karin Landers Joseph Gade Mitchell Gray Rémi Forest Mark Maxwell Roma Astemberg Thomas Gérard Jun Q. Yan Yue