Datasparq launches Houston: An open-source data pipeline orchestration tool
Houston, the tool to replace Airflow
Datasparq is pleased to announce the public beta launch of Houston, an open-source data pipeline orchestration tool designed to address the challenges of modern data infrastructure.
Visit the Houston website: https://www.callhouston.io/
Key features
- Microservices-based architecture: Enables modular development, reuse and language agnosticism, promoting flexibility and team collaboration.
- Serverless execution: Leverages cloud-native technologies for scalable and cost-effective deployments, eliminating infrastructure management overhead.
- Intuitive API and visual interface: Simplifies workflow definition and task management, reducing development time and complexity.
Who is Houston for?
- Data engineers: Seeking to streamline workflow, automate tasks and improve development efficiency.
- Data scientists: Requiring fast access to clean, prepared data for analysis and model training.
- DevOps teams: Responsible for maintaining reliable data pipelines within resource constraints.
How did Houston come to be?
In 2019, fuelled by a desire for simpler, more efficient data infrastructure, James Watkinson, Engineering Director at Datasparq, started to work on a serverless alternative to AirFlow. Taking inspiration from the moon landing missions, Houston was conceived as a data pipeline orchestration tool capable of bridging Google Cloud and Azure.
A dedicated team of five engineers contributed to Houston's growth, integrating it into critical data services for Datasparq's clients and solidifying its position as a cornerstone of Datasparq's engineering strategy. In 2023, the decision was made to rewrite Houston in Go (from its original Python base) and release it as open-source software. This move reflects Datasparq's commitment to the vibrant data engineering community.
Availability
Houston was publicly launched as an open source tool on Wednesday February 7th 2024.
Data professionals interested in exploring Houston can access the platform at https://callhouston.io.
