Franklin Obasipythocrypt.hashnode.net·Feb 20, 2023Creating Data Pipelines as DAGs in Apache Airflow (Part 1)A DAG is a special kind of graph. So, what exactly is a graph? A graph is used to express or illustrate relationships among objects. In more technical terms, graphs are used to describe any set of nodes and the edges (relationships) between the nodes...Discuss·1 likeapache
Anthonyanthzaro.hashnode.net·Oct 1, 2022Comparing GCP Cloud Composer environments with ease !Hey there ! If you need to diff two Cloud Composer environments, this tool is for you. cloudcomposerdiff is a python command line tool to diff two Cloud Composer environments. As per the example above, this tool looks at various Cloud Composer envir...Discuss·30 readsGCP
Ignacio Andrés Miranda Figueroaigna.hashnode.net·Jan 7, 2023Passing params from an Apache Airflow DAG to triggered DAGs using TriggerDagRunOperatorSo I was in this situation, struggling for like 5 hours yesterday (yes, the last 5 Friday work hours, the best ones to get stuck with some code) trying to pass parameters using the TriggerDagRunOperator, and wanting to die but at the end achieving it...Discuss·609 readsairflow
Karl Bolingerkbolinger.hashnode.net·Apr 17, 2023Building Data Pipelines with Apache Airflow: A Complete Guide with ExamplesData pipelines are a critical component of modern data infrastructures, allowing organizations to efficiently manage and process large volumes of data. Apache Airflow is an open-source platform that helps developers to create and manage data pipeline...DiscussData Engineering Basicsdata-engineering
Andrew Sharifikiaalireza-sharifikia.hashnode.net·Mar 17, 2023DataOps: Apache Airflow - BasicIntroduction Apache Airflow is an open-source platform for authoring, scheduling, and monitoring data and computing workflows. It was developed by Airbnb and is now under the Apache Software Foundation.It uses Python to create workflows that can be e...Discuss·46 readsDataOpsairflow
Arnold Kamanzikarnold.hashnode.net·Mar 3, 2023Automate DB Backup with Apache AirflowIf like me, you have administered a database and always find yourself tired of running the scripts to back up the database or even worse, you use a cronjob to automate the backup process, then something goes wrong and suddenly the backup fails and yo...Discuss·69 readsapache-airflow
Franklin Obasipythocrypt.hashnode.net·Feb 20, 2023Creating Data Pipelines as DAGs in Apache Airflow (Part 1)A DAG is a special kind of graph. So, what exactly is a graph? A graph is used to express or illustrate relationships among objects. In more technical terms, graphs are used to describe any set of nodes and the edges (relationships) between the nodes...Discuss·1 likeapache
Mike Kenneth Houngbadjimikekenneth.hashnode.net·Feb 4, 2023Building a Data Lakehouse for Analyzing Elon Musk Tweets using MinIO, Apache Airflow, Apache Drill and Apache SupersetEvery act of conscious learning requires the willingness to suffer an injury to one's self-esteem. That is why young children, before they are aware of their own self-importance, learn so easily.Thomas Szasz Motivation A Data Lakehouse is a modern d...Discussapache-airflow
Nikhil Chandrakarnikhilchandrakar.hashnode.net·Jan 16, 2023Basics of Apache airflowWhat is Apache Airflow? Why it is used? Apache airflow is an open-source platform for authoring, scheduling, monitoring data and computing workflows. It is a workflow management platform for batch-oriented workflows. It was started at Airbnb in Octob...Discussapache-airflow
Ignacio Andrés Miranda Figueroaigna.hashnode.net·Jan 7, 2023Passing params from an Apache Airflow DAG to triggered DAGs using TriggerDagRunOperatorSo I was in this situation, struggling for like 5 hours yesterday (yes, the last 5 Friday work hours, the best ones to get stuck with some code) trying to pass parameters using the TriggerDagRunOperator, and wanting to die but at the end achieving it...Discuss·609 readsairflow
Mike Kenneth Houngbadjimikekenneth.hashnode.net·Jan 6, 2023Twitter Data Pipeline with Apache Airflow + MinIO (S3 compatible Object Storage)Twitter Data Pipeline with Apache Airflow + MinIO (S3 compatible Object Storage) The more that you read, the more things you will know. The more that you learn, the more places you’ll go. Dr. Seuss Motivation During my journey as a Data Engineer, I...Discuss·38 readsTwitter
Anthonyanthzaro.hashnode.net·Oct 1, 2022Comparing GCP Cloud Composer environments with ease !Hey there ! If you need to diff two Cloud Composer environments, this tool is for you. cloudcomposerdiff is a python command line tool to diff two Cloud Composer environments. As per the example above, this tool looks at various Cloud Composer envir...Discuss·30 readsGCP
