This note covers two open-source tools. Kestra builds, schedules, and monitors complex workflows. Apache NiFi automates data pipelines for cybersecurity, observability, event streams, and generative AI. A browser UI designs, controls, and monitors workflows. Processing targets throughput and latency, with guaranteed delivery, retries, and horizontal scaling. Provenance tracking keeps a searchable history and lineage from source to destination. Plugin interfaces cover Processors and Controller Services, with Python processors and a REST API. Security uses OpenID Connect or SAML 2 single sign-on, role-based access control, and TLS or SFTP encryption. Java 21 is required. Python 3.10 or newer is optional. Uses include pipeline automation, ETL, real-time processing, and data distribution.
Kestra
- Kestra: An open-source platform for building, scheduling, and monitoring complex workflows - https://github.com/kestra-io/kestra
Apache NiFi
- Apache NiFi: An open-source data processing and distribution system that automates data pipelines for cybersecurity, observability, event streams, and generative AI - https://github.com/apache/nifi
- Key features:
- Browser UI: design, control, and monitor workflows through a web interface
- Scalable Processing: prioritizes throughput/latency, guarantees delivery with retries, horizontal scaling
- Provenance Tracking: searchable history, data lineage from source to destination
- Extensible Design: plugin interfaces for Processors and Controller Services, supports Python processors, REST API
- Security: single sign-on (OpenID Connect/SAML 2), role-based access control, TLS/SFTP encryption
- Requirements: Java 21, Python 3.10+ (optional)
- Use cases: Data pipeline automation, ETL workflows, real-time data processing, data distribution
- Key features:
See also: CI CD Tools · DevOps Tools