Use Prefect as a machine learning experiment tracker by leveraging mapping and artifacts. Are you familiar with the hottest new hyperparameter tuning framework called Prefect-ML? Well probably not, because it doesn’t exist. However, even though Prefect is primarily a workflow orchestration tool, it actually provides us with a lot…

Increase developer productivity and decrease costs for big data projects — An initial version of this article was published on James Le’s blog here. It has been updated to include new Fugue features. Fugue’s Motivation Data practitioners often start out by working with Pandas or SQL. Sooner or later, the size of data being processed outgrows what Pandas can handle efficiently, and distributed…

Run PyCaret functions on each partition of data distributedly — PyCaret is a low code machine learning framework that automates a lot of parts of the machine learning pipeline. With just a few lines of code, several models can be trained on a dataset. …

Increase developer productivity and decrease compute usage for big data projects — Fugue is an open-source library that provides a unified interface for Pandas, Spark, and Dask DataFrames. In this article, we will show how Fugue can accelerate the development of big data projects by decreasing compute costs and by increasing developer productivity. First, we’ll go over a simple code example that…

In a previous article, I wrote about FugueSQL as a SQL interface on top of Spark, Dask, and Pandas DataFrames. For those familiar with SparkSQL, FugueSQL is largely based on it, but with enhanced features to make it a first-class grammar for distributed computing workflows. When using SparkSQL, users often…

Note: This article is motivated by our experience working with Prefect community members deploying their Prefect workflows, but Prefect knowledge is not required to go through it. Outline Motivation Making a Python package Writing a Dockerfile Building and Using the Image Motivation Data scientists and analysts often shy away from packaging their…

We can do this the easy way, or the hard way — Note: The code presented is in images for formatting purposes but can all be found here. Motivation While Pandas is the most common Python tool for data science and data analyst workflows, it doesn’t scale as well to handling big datasets as it only uses one core at a time. It…

This tutorial will show how to bring Pandas libraries to Spark and Dask with Fugue. In this example, we’ll use the Pandera data validation library on Spark. — Data Validation Data validation is having checks in place to make sure that data comes in the format and specifications that we expect. As data pipelines become more interconnected, the chances of changes unintentionally breaking other pipelines also increase. Validations are used to guarantee that upstream changes will not break the integrity…

Seamlessly change between Pandas and Spark with one line of code — Motivation Data science teams often invest in a common set of tools and build a shared Python library. This library includes common ways to clean and transform data, as well as utility functions for machine learning (such as custom metrics or specific train-test splits). Having such a library provides a foundation…