SineKAN could outperform B-splines for specific Kolmogorov-Arnold Network tasks

Replacing B-splines with sinusoidal activation functions in Kolmogorov-Arnold Networks (KANs) completely transforms convergence dynamics. While the initial excitement surrounding KANs centered on spline flexibility, SineKAN investigates if periodic functions provide superior efficiency or generalization when approximating complex functions.

Why sinusoidal activations offer distinct advantages

Sinusoidal activations offer a distinct advantage. KANs work by replacing fixed node activations with learnable edge functions, typically implemented as B-splines. However, B-splines can be computationally demanding and often encounter difficulties extrapolating beyond their predefined grids. SineKAN swaps these for sine functions, essentially reconfiguring the network into a sequence of learnable frequency modulations.

This pivot is technically significant. Sinusoids are inherently designed to capture periodic patterns and possess a unique capacity to represent high-frequency components that might require many more parameters for splines to replicate. For AI workflows involving signal processing or physics-informed neural networks (PINNs), this method feels much more intuitive than attempting to force splines to fit wave patterns.

Regarding practical implementation and deployment, the primary difference from a standard KAN or MLP lies in the weight updates. The network optimizes the phase and frequency of the sine wave rather than updating a spline coefficient, which can accelerate convergence for specific mathematical functions.

How the architectural implementation logic works

The architectural implementation generally follows this logic:

1. The input undergoes multiplication by a learnable weight (frequency).
2. This resulting value passes through a $\sin(x)$ function.
3. A linear term or residual connection is frequently added to ensure stability and prevent periodicity-induced local minima.

Comparing SineKAN to Standard KANs reveals several key distinctions:

• Computational Overhead: SineKAN is typically leaner since calculating a sine function is less computationally expensive than evaluating a B-spline basis.
• Parameter Efficiency: Compared to the grid-based method used in original KANs, it often uses fewer parameters to represent oscillatory functions.
• Convergence Speed: It may converge more rapidly on periodic datasets, though it might struggle with monotonic functions that splines manage easily.
• Extrapolation: Sinusoids extrapolate periodically, serving as either a major advantage or a significant liability depending on the real-world data.

Where to find resources for building from scratch

The GitHub repository serves as a strong starting point for anyone attempting to build this from scratch. Because the math resides within the activation layer, swapping it into current KAN frameworks is relatively simple. It demonstrates how minor adjustments to an activation function can fundamentally change the performance of a regression model or an LLM agent.

https://arxiv.org/abs/2407.04149