Confluent and Apache Kafka (with Apache Flink)
What it is
Apache Kafka is the open-source distributed event-streaming log; Apache Flink is the open-source stateful stream processor; Confluent (now “IBM Confluent”) sells the commercial platform around both: Confluent Cloud, Confluent Platform, connectors, Schema Registry / stream governance, managed Flink, Tableflow, and Confluent Intelligence.
Maker, ownership and history
- Kafka created at LinkedIn (2008 per Wikipedia) by Jay Kreps, Jun Rao and Neha Narkhede, who founded Confluent on 2014-09-23. Kafka and Flink are Apache Software Foundation projects (Apache-2.0).
- Confluent filed for its IPO in June 2021 and traded on Nasdaq as CFLT; FY2024 revenue USD 963M, net loss USD 345M (Wikipedia, secondary).
- IBM acquisition: announced 2025-12-08 at about USD 11B; completed 2026-03-17 (Wikipedia; consistent with vault note IBM). ibm.com now sells the product as “IBM Confluent”. IBM press release pages were not reachable (404), so deal terms beyond these are unverified.
Editions and deployment
Confluent Cloud (cluster types Basic, Standard, Enterprise, Freight; consumption billing by eCKU hour, data in/out and storage), Confluent Platform (self-managed, free on a single broker, 30-day trial otherwise), Confluent Private Cloud (hybrid) and WarpStream (private-cloud alternative), per ibm.com/products/confluent. Pricing: https://www.confluent.io/confluent-cloud/pricing/. Apache Kafka itself is free. Latest Kafka release found: 4.3.1 (2026-06-25); Flink 2.3.0 (2026-06-25).
Core architecture
Kafka: partitioned append-only log, replicated brokers, consumer groups, Kafka Connect and Kafka Streams; Kafka 4.0.0 was released 2025-03-18 (downloads page). Flink: SQL and DataStream APIs, exactly-once state. Tableflow materialises topics as Iceberg tables so streams land in the lakehouse without a separate ETL job.
Role in an enterprise AI rollout
Event backbone for freshness: operational events, CDC and clickstream reach the lakehouse, feature stores, vector indexes and agents in seconds. Streaming agents and real-time context replace nightly batch for use cases such as fraud, personalisation and agent triggers. See data-pipelines-for-ai.
AI features as of October 2026
Confluent blog and ibm.com (fetched 2026-10-07) list: Confluent Intelligence (launched 2025), Streaming Agents, Flink UDF/SQL enhancements, IBM Granite Time Series and TimesFM model support, Tableflow, and an open-source Confluent MCP server plus Agent Skills described as generally available. A watsonx.data pairing for governed real-time context is announced on ibm.com.
Integrations
Iceberg / Delta via Tableflow, Snowflake, Databricks, 150 pre-built connectors, Debezium-style CDC, IBM watsonx.data. Complement to ingestion tools such as fivetran and airbyte (batch/CDC) and orchestrators (apache-airflow).
Strengths and weaknesses (opinion)
Strong: de-facto streaming standard, rich governance, managed Flink. Weak: operational and cost complexity at scale; commercial roadmap now tied to IBM; Kafka is overkill for batch-only needs. Alternatives (Redpanda, Amazon MSK, Azure Event Hubs) not researched here.
Self-learning
- Apache Kafka docs: https://kafka.apache.org/documentation/ (free)
- Apache Flink: https://flink.apache.org/ (free)
- Confluent Developer courses: https://developer.confluent.io/courses/ (free)
- Confluent learn: https://www.confluent.io/learn/ (free)
- Certifications: not verified.
Sources (fetched 2026-10-07)
- https://en.wikipedia.org/wiki/Confluent (secondary)
- https://www.ibm.com/products/confluent ; https://www.confluent.io/product/ ; https://www.confluent.io/blog/ ; /confluent-cloud/pricing/
- https://kafka.apache.org/community/downloads/ ; https://flink.apache.org/downloads/
Open items
- IBM deal terms and close confirmed 2026-10-07 from the IBM newsroom: announced 2025-12-08, completed 2026-03-17, 11 billion (https://newsroom.ibm.com/2026-03-17-IBM-Completes-Acquisition-of-Confluent,-Making-Real-Time-Data-the-Engine-of-Enterprise-AI-and-Agents).
- Confluent MCP server repo and GA date; Streaming Agents / Intelligence tier limits.