Sparse AutoEncoder: from Superposition to interpretable features - Related to time, interpretable, superposition, marketing, sparse

How to Find Seasonality Patterns in Time Series

In my professional life as a data scientist, I have encountered time series multiple times. Most of my knowledge comes from my academic experience, specifically my courses in Econometrics (I have a degree in Economics), where we studied statistical properties and models of time series.

Among the models I studied was SARIMA, which acknowledges the seasonality of a time series, however, we have never studied how to intercept and recognize seasonality patterns.

Most of the time I had to find seasonal patterns I simply relied on visual inspections of data. This was until I stumbled on this YouTube video on Fourier transforms and eventually found out what a periodogram is.

In this blog post, I will explain and apply simple concepts that will turn into useful tools that every DS who’s studying time series should know.

What is a Fourier Transform? Fourier Transform in Python Periodogram.

Let’s assume I have the following dataset (AEP energy consumption, CC0 license):

import pandas as pd import [website] as plt df = pd.read_csv("data/[website]", index_col=0) [website] = pd.to_datetime([website] df.sort_index(inplace=True) fig, ax = plt.subplots(figsize=(20,4)) [website] plt.tight_layout() [website].

AEP hourly energy consumption | Image by Author.

It is very clear, just from a visual inspection, that seasonal patterns are playing a role, however it might be trivial to intercept them all.

As explained before, the discovery process I used to perform was mainly manual, and it could have looked something as follows:

fig, ax = plt.subplots(3, 1, figsize=(20,9)) df_3y = df[([website] >= '2006–01–01') & ([website] < '2010–01–01')] df_3M = df[([website] >= '2006–01–01') & ([website] < '2006–04–01')] df_7d = df[([website] >= '2006–01–01') & ([website] < '2006–01–08')] ax[0].set_title('AEP energy consumption 3Y') df_3y[['AEP_MW']].groupby(pd.Grouper(freq = 'D')).sum().plot(ax=ax[0]) for date in df_3y[[True if x % (24 * [website] / 2) == 0 else False for x in range(len(df_3y))]][website] ax[0].axvline(date, color = 'r', alpha = [website] ax[1].set_title('AEP energy consumption 3M') df_3M[['AEP_MW']].plot(ax=ax[1]) for date in df_3M[[True if x % (24 * 7) == 0 else False for x in range(len(df_3M))]][website] ax[1].axvline(date, color = 'r', alpha = [website] ax[2].set_title('AEP energy consumption 7D') df_7d[['AEP_MW']].plot(ax=ax[2]) for date in df_7d[[True if x % 24 == 0 else False for x in range(len(df_7d))]][website] ax[2].axvline(date, color = 'r', alpha = [website] plt.tight_layout() [website].

AEP hourly energy consumption, smaller timeframe | Image by Author.

This is a more in-depth visualization of this time series. As we can see the following patterns are influencing the data: **- a 6 month cycle,.

This dataset demonstrates energy consumption, so these seasonal patterns are easily inferable just from domain knowledge. However, by relying only on a manual inspection we could miss crucial informations. These could be some of the main drawbacks:

Subjectivity: We might miss less obvious patterns.

We might miss less obvious patterns. Time-consuming : We need to test different timeframes one by one.

: We need to test different timeframes one by one. Scalability issues: Works well for a few datasets, but inefficient for large-scale analysis.

As a Data Scientist it would be useful to have a tool that gives us immediate feedback on the most significant frequencies that compose the time series. This is where the Fourier Transforms come to help.

The Fourier Transform is a mathematical tool that allows us to "switch domain".

Usually, we visualize our data in the time domain. However, using a Fourier Transform, we can switch to the frequency domain, which demonstrates the frequencies that are present in the signal and their relative contribution to the original time series.

Any well-behaved function f(x) can be written as a sum of sinusoids with different frequencies, amplitudes and phases. In simple terms, every signal (time series) is just a combination of simple waveforms.

F(f) represents the function in the frequency domain .

. f(x) is the original function in the time domain .

. exp(−i2πf(x)) is a complex exponential that acts as a "frequency filter".

Thus, F(f) tells us how much frequency f is present in the original function.

Let’s consider a signal composed of three sine waves with frequencies 2 Hz, 3 Hz, and 5 Hz:

A Simple Signal in time domain | Image by Author.

Now, let’s apply a Fourier Transform to extract these frequencies from the signal:

A Simple Signal in the frequency domain | Image by Author.

The graph above represents our signal expressed in the frequency domain instead of the classic time domain. From the resulting plot, we can see that our signal is decomposed in 3 elements of frequency 2 Hz, 3 Hz and 5 Hz as expected from the starting signal.

As noted before, any well-behaved function can be written as a sum of sinusoids. With the information we have so far it is possible to decompose our signal into three sinusoids:

A Simple Signal decomposition in its basic wavelength | Image by Author.

The original signal (in blue) can be obtained by summing the three waves (in red). This process can easily be applied in any time series to evaluate the main frequencies that compose the time series.

Given that it is quite easy to switch between the time domain and the frequency domain, let’s have a look at the AEP energy consumption time series we started studying at the beginning of the article.

Python provides the "[website]" library to compute the Fourier Transform for discrete signals. FFT stands for Fast Fourier Transform which is an algorithm used to decompose a discrete signal into its frequency components:

from numpy import fft X = [website]['AEP_MW']) N = len(X) frequencies = fft.fftfreq(N, 1) periods = 1 / frequencies fft_magnitude = [website] / N mask = frequencies >= 0 # Plot the Fourier Transform fig, ax = plt.subplots(figsize=(20, 3)) [website][mask], fft_magnitude[mask]) # Only plot positive frequencies ax.set_xscale('log') [website]'{x:,.0f}') ax.set_title('AEP energy consumption - Frequency-Domain') ax.set_xlabel('Frequency (Hz)') ax.set_ylabel('Magnitude') [website].

AEP hourly energy consumption in frequency domain | Image by Author.

This is the frequency domain visualization of the AEP_MW energy consumption. When we analyze the graph we can already see that at certain frequencies we have a higher magnitude, implying higher importance of such frequencies.

However, before doing so we add one more piece of theory that will allow us to build a periodogram, that will give us a advanced view of the most essential frequencies.

The periodogram is a frequency-domain representation of the power spectral density (PSD) of a signal. While the Fourier Transform tells us which frequencies are present in a signal, the periodogram quantifies the power (or intensity) of those frequencies. This passage is usefull as it reduces the noise of less essential frequencies.

Mathematically, the periodogram is given by:

P(f) is the power spectral density (PSD) at frequency f,.

X(f) is the Fourier Transform of the signal,.

This can be achieved in Python as follows:

power_spectrum = [website]**2 / N # Power at each frequency fig, ax = plt.subplots(figsize=(20, 3)) [website][mask], power_spectrum[mask]) ax.set_title('AEP energy consumption Periodogram') ax.set_xscale('log') [website]'{x:,.0f}') [website]'Frequency (Hz)') [website]'Power') [website].

AEP hourly energy consumption Periodogram | Image by Author.

From this periodogram, it is now possible to draw conclusions. As we can see the most powerful frequencies sit at:

[website] Hz, corresponding to 6 months,.

and at 168 Hz, corresponding to the weekly cycle.

These three are the same Seasonality components we found in the manual exercise done in the visual inspection. However, using this visualization, we can see three other cycles, weaker in power, but present:

an 84 Hz cycle, correspondint to half a week,.

an [website] Hz cycle, corresponding to a full year.

It is also possible to use the function "periodogram" present in scipy to obtain the same result.

from [website] import periodogram frequencies, power_spectrum = periodogram(df['AEP_MW'], return_onesided=False) periods = 1 / frequencies fig, ax = plt.subplots(figsize=(20, 3)) [website], power_spectrum) ax.set_title('Periodogram') ax.set_xscale('log') [website]'{x:,.0f}') [website]'Frequency (Hz)') [website]'Power') [website].

When we are dealing with time series one of the most key components to consider is seasonalities.

In this blog post, we’ve seen how to easily discover seasonalities within a time series using a periodogram. Providing us with a simple-to-implement tool that will become extremely useful in the exploratory process.

Please leave some claps if you enjoyed the article and feel free to comment, any suggestion and feedback is appreciated!

_Here you can find a notebook with the code from this blog post._.

Research Google DeepMind at NeurIPS 2024 Share.

Building adaptive, smart, and safe AI Agents LLM-based AI agents are showing promis...

Research AI for the board game Diplomacy Share.

Agents cooperate advanced by communicating and negotiating, and sanctioning broken pr...

What's improved than an AI chatbot that can assist you with tasks? One that can do them for you. OpenAI continues to build out its AI agents in ...

Sparse AutoEncoder: from Superposition to interpretable features

Sparse AutoEncoder: from Superposition to interpretable aspects.

Complex neural networks, such as Large Language Models (LLMs), suffer quite often from interpretability challenges. One of the most significant reasons for such difficulty is superposition — a phenomenon of the neural network having fewer dimensions than the number of capabilities it has to represent. For example, a toy LLM with 2 neurons has to present 6 different language capabilities. As a result, we observe often that a single neuron needs to activate for multiple capabilities. For a more detailed explanation and definition of superposition, please refer to my previous blog post: “Superposition: What Makes it Difficult to Explain Neural Network”.

In this blog post, we take one step further: let’s try to disentangle some fsuperposed elements. I will introduce a methodology called Sparse Autoencoder to decompose complex neural network, especially LLM into interpretable elements, with a toy example of language elements.

A Sparse Autoencoder, by definition, is an Autoencoder with sparsity introduced on purpose in the activations of its hidden layers. With a rather simple structure and light training process, it aims to decompose a complex neural network and uncover the capabilities in a more interpretable way and more understandable to humans.

Let us imagine that you have a trained neural network. The autoencoder is not part of the training process of the model itself but is instead a post-hoc analysis tool. The original model has its own activations, and these activations are collected afterwards and then used as input data for the sparse autoencoder.

For example, we suppose that your original model is a neural network with one hidden layer of 5 neurons. Besides, you have a training dataset of 5000 samples. You have to collect all the values of the 5-dimensional activation of the hidden layer for all your 5000 training samples, and they are now the input for your sparse autoencoder.

Image by author: Autoencoder to analyse an LLM.

The autoencoder then learns a new, sparse representation from these activations. The encoder maps the original MLP activations into a new vector space with higher representation dimensions. Looking back at my previous 5-neuron simple example, we might consider to map it into a vector space with 20 aspects. Hopefully, we will obtain a sparse autoencoder effectively decomposing the original MLP activations into a representation, easier to interpret and analyze.

Sparsity is an crucial in the autoencoder because it is necessary for the autoencoder to “disentangle” functions, with more “freedom” than in a dense, overlapping space.. Without existence of sparsity, the autoencoder will probably the autoencoder might just learn a trivial compression without any meaningful functions’ formation.

Let us now build our toy model. I beg the readers to note that this model is not realistic and even a bit silly in practice but it is sufficient to showcase how we build sparse autoencoder and capture some functions.

Suppose now we have built a language model which has one particular hidden layer whose activation has three dimensions. Let us suppose also that we have the following tokens: “cat,” “happy cat,” “dog,” “energetic dog,” “not cat,” “not dog,” “robot,” and “AI assistant” in the training dataset and they have the following activation values.

[[website], [website], [website], [website]], # "cat"

[[website], [website], [website], [website]], # "happy cat" (similar to "cat").

[[website], [website], [website], [website]], # "dog"

[[website], [website], [website], [website]], # "loyal dog" (similar to "dog").

[[website], [website], [website], [website]], # "not cat"

[[website], [website], [website], [website]], # "not dog"

# Robot and AI assistant (more distinct in 4D space).

[[website], [website], [website], [website]], # "robot"

[[website], [website], [website], [website]] # "AI assistant"

We now build the autoencoder with the following code:

def __init__(self, input_dim, hidden_dim):

super(SparseAutoencoder, self).__init__().

, we see that the encoder has a only one fully connected linear layer, mapping the input to a hidden representation with hidden_dim and it then passes to a ReLU activation. The decoder uses just one linear layer to reconstruct the input. Note that the absence of ReLU activation in the decoder is intentional for our specific reconstruction case, because the reconstruction might contain real-valued and potentially negative valued data. A ReLU would on the contrary force the output to stay non-negative, which is not desirable for our reconstruction.

We train model using the code below. Here, the loss function has two parts: the reconstruction loss, measuring the accuracy of the autoencoder’s reconstruction of the input data, and a sparsity loss (with weight), which encourages sparsity formulation in the encoder.

reconstruction_loss = criterion(decoded, data).

# Sparsity penalty (L1 regularization on the encoded aspects).

loss = reconstruction_loss + sparsity_weight * sparsity_loss.

Now we can have a look of the result. We have plotted the encoder’s output value of each activation of the original models. Recall that the input tokens are “cat,” “happy cat,” “dog,” “energetic dog,” “not cat,” “not dog,” “robot,” and “AI assistant”.

Image by author: attributes learned by encoder.

Even though the original model was designed with a very simple architecture without any deep consideration, the autoencoder has still captured meaningful elements of this trivial model. , we can observe at least four elements that appear to be learned by the encoder.

Give first Feature 1 a consideration. This feautre has big activation values on the 4 following tokens: “cat”, “happy cat”, “dog”, and “energetic dog”. The result implies that Feature 1 can be something related to “animals” or “pets”. Feature 2 is also an interesting example, activating on two tokens “robot” and “AI assistant”. We guess, therefore, this feature has something to do with “artificial and robotics”, indicating the model’s understanding on technological contexts. Feature 3 has activation on 4 tokens: “not cat”, “not dog”, “robot” and “AI assistant” and this is possibly a feature “not an animal”.

Unfortunately, original model is not a real model trained on real-world text, but rather artificially designed with the assumption that similar tokens have some similarity in the activation vector space. However, the results still provide interesting insights: the sparse autoencoder succeeded in showing some meaningful, human-friendly functions or real-world concepts.

The simple result in this blog post points to:, a sparse autoencoder can effectively help to get high-level, interpretable capabilities from complex neural networks such as LLM.

For readers interested in a real-world implementation of sparse autoencoders, I recommend this article, where an autoencoder was trained to interpret a real large language model with 512 neurons. This study provides a real application of sparse autoencoders in the context of LLM’s interpretability.

Finally, I provide here this google colab notebook for my detailed implementation mentioned in this article.

How to Make a Data Science Portfolio That Stands Out.

My website that we are are going to create.

Many people have asked how I made my website. In th......

'ZDNET Recommends': What exactly does it mean?

ZDNET's recommendations are based on many hours of testing, research, and comparison shopping. We gath......

Meta CEO Mark Zuckerberg says this is the year artificial intelligence will start to make possible autonomous software engineering "......

New-Generation Marketing Mix Modelling with Meridian

Let’s now use the Meridian library with data. The first step is to install Meridian with either pip or poetry : pip install google-meridian or poetry add google-meridian.

We will then get the data and start defining columns which are of interest to us.

For the control variables, we will use all of the holidays variables in the dataset. Our KPI will be sales, and the time granularity will be weekly.

Next, we will select our Media variables. Meridian makes a difference between media data and media spends:

Media data (or “execution”) : Contains the exposure metric per channel and time span (such as impressions per time period). Media values must not contain negative values. When exposure metrics are not available, use the same as in media spend.

Media spend : Containing the media spending per channel and time span. The media data and media spend must have the same dimensions.

When should you use spends vs execution ?

It is usually recommended to use exposure metrics as direct inputs into the model as they represent how media activity has been consumed by consumers. However, no one plans a budget using execution data. If you use MMM to optimize budget planning, my advice would be to use data you control, ie spends.

In our use case, we will only use the spends from 5 channels: Newspaper, Radio, TV, Social Media and Online Display.

CONTROL_COLS = [col for col in raw_df.columns if 'hldy_' in col].

data_df = raw_df[[DATE_COL, SALES_COL, *MEDIA_COLS, *CONTROL_COLS]].

data_df[DATE_COL] = pd.to_datetime(data_df[DATE_COL]).

We will then map the columns to their data type so that Meridian can understand them. The CoordToColumns object will help us do that, and requires mandatory information :

time : the time column (usually a date, day or week).

: the time column (usually a date, day or week) controls : the control variables.

: the control variables kpi : the response we want the model to predict. In our case, we will give it the value revenue since we want to predict sales.

There several other parameters which can be used, namely the geo parameter if we have several groups (geographies for ex.), population , reach , frequency . Details about these are out of this scope but the documentation can be found here.

We can therefore create our column mappings :

Next, we will use our dataframe and the columns mappings to create a data object to be used by the model.

data_df.set_index("wk_strt_dt")[SALES_COL].plot(color=COLORS[1], ax=ax).

[website]"Sales", xlabel='date', ylabel="sales");

There seems to be a nice seasonality with peaks around Christmas. Trend is overall constant with a level oscillating between 50 and 150M.

fig, ax = plt.subplots(5, figsize=(20,30)).

for axis, channel in zip(ax, spends_columns_raw):

data_df.set_index("wk_strt_dt")[channel].plot(ax=axis, color=COLORS[1]).

[website][channel], xlabel="Date", ylabel="Spend");

We observe a clearly decreasing trend for newspaper correlated with an increasing trend for Social Media. Spends seem to be also increasing at or just before Christmas.

Building the model and choosing the right parameters can be quite complex as there are a lot of options available. I will share here my findings but feel free to explore by yourself.

The first part is to choose the priors for our media spends. We will use the PriorDistribution class which allows us to define several variables. You can change the priors of almost any parameter of the model (mu, tau, gamma, beta, etc…), but for now we will only focus on the beta which are the coefficients of our media variables. My recommendation is, if you are using spends only, to use the beta_m . You can choose the roi_m or mroi_m but you will need to adapt the code to use a different prior.

from [website] import prior_distribution.

prior = prior_distribution.PriorDistribution(.

# If you want to use the ROI vision instead of the coefficients approach.

When defining the model specifications, you will then be able to define :

max_len : the maximum number of lag periods (≥ `0`) to.

include in the Adstock calculation. I recommend choosing between 2 and 6.

: the maximum number of lag periods (≥ `0`) to include in the Adstock calculation. I recommend choosing between 2 and 6. paid_media_prior_type : if you choose to model the beta_m , then choose coefficient . Else, choose roi or mroi .

: if you choose to model the , then choose . Else, choose or . knots : Meridian applies automatic seasonality adjustment through a time-varying intercept approach, controlled by the knots value. You can set a value of 1 (constant intercept, no seasonality modelling), or equal to a given number that must be lower than the length of the data. A low value could lead to a low baseline, a high value could lead to overfitting and lead to a baseline eating everything. I recommend to set it to 10% of the number of data points.

It is also possible to define a train-test split to avoid overfitting via the holdout_id parameter. I won’t cover it here, but it is a best practice to have this split done for model selection.

mmm = model.Meridian(input_data=data, model_spec=model_spec).

Fitting the model can be slow if you have a large number of data points and variables. I recommend to start with 2 chains, and leave the default number of samples:

mmm.sample_posterior(n_chains=2, n_adapt=500, n_burnin=500, n_keep=1000).

Once the model is done running, we will perform a series of checks to ensure that we can use it confidently.

R-hat close to [website] indicate convergence. R-hat < [website] indicates approximate convergence and is a reasonable threshold for many problems.

A lack of convergence typically has one of two culprits. Either the model is very poorly misspecified for the data, which can be in the likelihood (model specification) or in the prior. Or, there is not enough burnin, meaning n_adapt + n_burnin is not large enough.

from meridian.analysis import visualizer.

model_diagnostics = visualizer.ModelDiagnostics(mmm).

We see that all r-hat values are below [website], which indicates no divergence or issue during training.

The model trace contains the sample values from the chains. A nice trace is when the two posterior distributions (as we have 2 chains) for a given parameter overlap nicely. In the diagram below, you can see that blue and black lines on the left-hand side nicely overlap :

during fitting, we will compare prior vs posterior distribution. If they perfectly overlap, this means that our model has not shifted its prior distributions and therefore has probably not learned anything, or that the priors were misspecified., we would like to see a slight shift in distributions :

We clearly that that the priors and posteriors don’t overlap. For TV and Social Media for ex, we see that the orange HalfNormal priors have shifted to the blue quasi-Normal distributions.

Finally, we will use metrics to evaluate our model fit. You probably know about metrics like R2, MAPE, etc., so let’s have a look at those values:

model_diagnostics = visualizer.ModelDiagnostics(mmm).

model_diagnostics.predictive_accuracy_table().

Obviously, a R2 of [website] is not great at all. We could improve that by either adding more knots in the baseline, or more data to the model, or play with the priors to try to capture more information.

Remember that one of the objectives of MMM is to provide you with media contributions vs your sales. This is what we will look at with a waterfall diagram :

media_summary = visualizer.MediaSummary(mmm).

media_summary.plot_contribution_waterfall_chart().

What we usually expect is to have a baseline between 60 and 80%. Keep in mind that this value can be very sensitive and depend on the model specification and parameters. I encourage you to play with different knots values and priors and see the impact it can have on the model.

The spend versus contribution chart compares the spend and incremental revenue or KPI split between channels. The green bar highlights the return on investment (ROI) for each channel.

We see that the highest ROI comes from Social Media, followed by TV. But this is also where the uncertainty interval is the largest. MMM is not an exact answer : it gives you values AND uncertainty associated to those. My opinion here is that uncertainty intervals are very large. Maybe we should use more sampling steps or add more variables to the model.

Remember that one of the objectives of the MMM is to propose an optimal allocation of spends to maximize revenue. This can be done first by looking at what we call response curves. Response curves describe the relationship between marketing spend and the resulting incremental revenue.

incremental revenue increases as the spend increases for some touchpoints like newspaper, growth is slower, meaning a 2x increase in spend will not translate to a 2x incremental revenue.

The goal of the optimization will be to take those curves and navigate to find the best combination of value that maximize our sales equation. We know that sales = f(media, control, baseline), and we are trying to find the media* values that maximize our function.

We can choose between several optimization problems, for ex:

How can I reach the sames sales level with less budget ?

Given the same budget, what is the maximum revenue I can get ?

Let’s use Meridian to optimize our budget and maximize sales (scenario 1). We will use the default parameters here but it is possible to fine-tune the constraints on each channel to limit the search scope.

budget_optimizer = optimizer.BudgetOptimizer(mmm).

optimization_results = budget_optimizer.optimize().

# Plot the response curves before and after.

optimization_results.plot_response_curves().

We can see that the optimizer recommends to decrease the spends for Newspaper, Online Display and recommends to increase spends for Radio, Social Media and TV.

How does it translate in terms of revenue ?

3% increase in revenue just by rebalancing our budget ! Of course this conclusion is a bit hasty. First, replaying the past is easy. You have no guarantee that your baseline sales (60%) would behave the same next year. Think of Covid. Second, our model does not account for interactions between channels. What we have used here is a simple additional model, but some approaches use a log-log multiplicative model to account for interactions between variables. Third, there is uncertainty in our response curves which is not handled by the optimizer, as it only takes the average response curve for each channel. Response curves with uncertainty look like the picture below and optimizing under uncertainty becomes a lot more complex :

However, it still gives you an idea of where you are maybe over or under-spending.

MMM is a complex but powerful tool that can uncover insights from your marketing data, help you understand your marketing efficiency and assist you in budget planning. The new methods relying on Bayesian inference provide nice feature such as adstock and saturation modelling, incorporation of geographic-level data, uncertainty levels and optimization capabilities. Happy coding.

The next time you use Gemini, you might notice it's a little faster.

Google unveiled that Gemini [website] Flash AI is now rolling......

PETER CATTERALL/Contributor/Getty Images.

Barely a week into its new-found fame, DeepSeek -- and the story about its development -- is evolving at bre......

DeepSeek V3: A New Contender in AI-Powered Data Science.

How DeepSeek’s budget-friendly AI model stacks up against ChatGPT, Claude, and Gemini in SQL,......