All categorical values are displayed in the legend when source dataframe is polars.DataFrame

[knl]

This is the example that produces all categorical values in the legend, which is unexpected. I copied it from a notebook, so might need tweaking to display elsewhere.

import polars as pl
import polars.selectors as cs
import datetime as dt
import random

import plotnine
from plotnine import *

import mizani
from mizani.formatters import log_format, custom_format, date_format, label_number, label_percent, label_bytes
from mizani.breaks import date_breaks

# turn on String Cache, otherwise we can't concat different dataframes into one
pl.enable_string_cache()


date = dt.datetime(2025, 8, 29)          # any date is fine
start = dt.datetime(date.year, date.month, date.day, 6, 0, 0)
end   = dt.datetime(date.year, date.month, date.day, 18, 0, 0)

# timestamps: every 5 minutes, inclusive of both ends (06:00 and 18:00)
timestamps = pl.datetime_range(
    start=start,
    end=end,
    interval="5m",
    closed="both",
    eager=True
)

random.seed(42)

sources = pl.Series("source", ["u", "w", "v"], dtype=pl.Categorical)
dirs = pl.Series("dir", ["incoming", "outgoing"], dtype=pl.Categorical)

df = (
    pl.DataFrame({"source": sources})
    .join(
        pl.DataFrame({"five_min_slot": timestamps}),
        how="cross"
    )
    .join(
        pl.DataFrame({"dir": dirs}),
        how="cross"
    )
    .with_columns(
        values=pl.lit(100.0),
    )
    .with_columns(
        pl.col('values').map_elements(lambda v: v*random.random(), return_dtype=pl.self_dtype()),
    )
)

(
    ggplot(
        df.filter(
            pl.col('source').eq('u')
        )
        .sort('five_min_slot'),
        aes(x='five_min_slot', y='values', color='factor(dir)')
    )
    + geom_line()
    + theme_linedraw()
    + theme(
        figure_size=(16, 5),
        axis_text_x=element_text(rotation=45, ha='right'),
    )
    + scale_x_datetime(
        labels=date_format("%H:%M"),
        date_breaks="15 minute",
        date_minor_breaks="5 minute",
        expand=(0, 0),
    )
    + scale_y_continuous(
        labels=label_percent(scale=1),
    )

)

For that code, I get the following output:

I would expect the legend to contain only incoming and outgoing.

[TyberiusPrime]

For a Categorical, the assumption is you want all categories - see for example pandas.DataFrame.value_counts() which will include all levels as well.

Let me turn this around, conceptually:

If plotnine wasn't showing all levels in the legend,
how would you produce multiple graphs that shared the same legend (and color assignment), even if each had just had a subset of the data?

By setting 'breaks' on a scale_color.

You'll either need to modify your categorical, or set breaks on a scale_color.

[knl]

Sorry for not being clear, but the series I’m plotting should only have two categories, not 5. With pandas, categories are “constrained” to the given series. Because of the way how categorical cache works in polars, categorical values are global. Thus, if I would have another series with 1000 categorical levels, it would show up in this graph, and that is unexpected.

[TyberiusPrime]

My apologies, I missed that fine but important detail.

[knl]

I did a loop over several polars and ggplot versions, and this issue seems to come from polars 1.32. All previous versions display two categorical levels. This PR might be causing the issues: pola-rs/polars#23016

[has2k1]

I think polars has totally changed the meaning of categoricals. They are no longer independent!

Someone else has just filed an issue.

[knl]

Polars categoricals are indeed becoming weirder. However, there is another data type in there that fills the role categoricals had before: Enum. I noticed that when using Enum column in a facet, the ordering is not taken into the account. If someone points me in the right direction, I can try adding Enum support to plotnine.

[has2k1]

According to polars, Enums are the equivalent of _Ordered Categorical_s, atleast that is what polars translates them to pandas as and we do accept that.
That leaves the polars Categorical to be the Unordered Categorical but with features and an API that we expect of unordered categoricals because that makes them useful.

We expect of unordered categoricals:

1. They should have a listing order.
2. They should be sortable according to the listing order.
3. The listing order should be "setable" (it is okay if the process creates another categorical and returns it, rather than do an actual mutation).
4. The domain of a categorical variable should be local to the variable. It should not be shared with other categoricals in the dataframe or globally.

Now, the "listing order" being minded in 1, 2 & 3 does not turn an Unordered Categorical into an Ordered one, rather that is the information on how to present the Unordered Categorical.