↓ Skip to main content
  1. Posts/

Cacimbão: Brazilian datasets for learning, built with Polars and Narwhals

Table of Contents

During my data visualization classes at FU I noticed, once again, a pattern that has been around in the data field for a while: a lot of the data used in classes and tutorials is from the United States. Before you know it, we’re analyzing what people over there watch on Netflix, California house prices, and even how votes are distributed. As a Brazilian, none of that really makes much sense to me.

Brazil has an open data policy that’s a reference worldwide. The default is that data is public, unless there’s a good justification. That’s why we have thousands of datasets to explore. DATASUS, for example, is the largest open clinical dataset in the world.

We have an abundance of data in many domains to explore. You can learn about X, Y or Z using national, state, and even city-level data. There’s just one problem: for each dataset, you need to download the files, load them, and only then start having fun. For data mostly coming from the US, that’s already done by popular libraries like scikit-learn datasets and vega datasets, and by sites like Kaggle and OpenML.

With that in mind, I created cacimbão, a library of real Brazilian datasets for learning. Focused on Brazil, with Brazilian data and tuned for the Brazilian reality.

How to use
#

One of the strong points, and I believe what led other libraries to take off, is ease of use. With cacimbão it’s no different. To install, run:

pip install cacimbao

Then, you can import the library to see the available datasets:

import cacimbao

cacimbao.list_datasets()

If you’d like to see more details about the available datasets, you can use:

cacimbao.list_datasets(include_metadata=True)

Loading a dataset
#

You can do this by loading a local dataset (no internet needed):

df = cacimbao.load_dataset("pescadores_e_pescadoras_profissionais")

And also by loading a remote dataset (where we download the data from the internet):

df = cacimbao.download_dataset("filmografia_brasileira")

You can check whether a dataset is local or not via the metadata (cacimbao.list_datasets(include_metadata=True)).

Having local/remote datasets and knowing the dataset sizes helps in cases of limited internet access and when you want to keep machine resource usage low.

Okay, but what are Polars and Narwhals?
#

For the nerdier folks, here’s something cool I’ve been experimenting with in this project. Lately I’ve been using Polars, the data community’s latest darling. But I know many people still use Pandas and I’d like to keep this library as accessible as possible.

That’s where Narwhals comes in! 🐋 Narwhals is a library that provides an interface between different dataframes, like Pandas, Polars, Duckdb, and others. That way I can work with Cacimbão’s datasets using a Polars dataframe while other people can keep using Pandas the same way. By default, the returned dataframe is Polars. If you’d like to use Polars too (come on, it’s great!), use the df_format parameter:

df = cacimbao.download_dataset("filmografia_brasileira", df_format="pandas")
type(df)
df = cacimbao.download_dataset("filmografia_brasileira", df_format="polars")
type(df)

Next steps
#

Currently the library has 5 datasets across the areas of culture, health, economy, labor, and Indigenous peoples. I still want to add more datasets but also create notebooks showing how to build data visualizations, exploratory analyses, and machine learning models with Cacimbão.

Who knows, maybe we’ll move on from decision trees on the Titanic to diagnoses with data from our National Health Survey. Data, we’ve got!