Physics-Informed Neural Networks (PINNs)

Prerequisites

• Module 3.1: Neural Differential Equations (neural network basics, automatic differentiation)
• Module 2.1: Scalar PDEs and Classification (PDE formulation, boundary conditions)
• Familiarity with gradient descent and chain rule

Why This Matters

Traditional numerical PDE solvers (finite differences, finite elements) require meshes, careful discretization, and separate code for each equation. Physics-Informed Neural Networks (PINNs) offer a radically different approach: approximate the solution with a neural network and train it by minimizing the PDE residual at randomly sampled collocation points. The physics itself becomes the loss function.

This means PINNs can solve forward problems (given equation and conditions, find the solution), inverse problems (given partial observations, recover unknown parameters), and even problems on irregular domains without mesh generation. Applications span fluid dynamics, heat transfer, solid mechanics, medical imaging, and geophysics. PINNs are not a replacement for classical solvers but a powerful complement, especially when data and physics must be combined.

Learning Objectives

1. Formulate the PINN loss function from a PDE, initial conditions, and boundary conditions.
2. Explain how automatic differentiation computes $\partial u/\partial t$, $\partial^2 u/\partial x^2$ through the network.
3. Sample collocation points and construct the training set for a PINN.
4. Implement a PINN for a forward ODE problem in pure NumPy (toy scale).
5. Understand the role of loss weighting ($\lambda_{\text{PDE}}$, $\lambda_{\text{BC}}$, $\lambda_{\text{IC}}$) and its impact on training.
6. Set up an inverse problem where an unknown physical parameter is learned alongside the solution.
7. Analyze PINN failure modes: spectral bias, causality violation, loss imbalance.
8. Compare PINNs with classical finite-difference solvers on accuracy and computational cost.
9. Frame collocation point selection as an active-learning agent problem.

Core Concepts

Common Misconceptions

Worked Examples

Interactive Code Lab

import numpy as np
import matplotlib.pyplot as plt

# ── Tiny MLP in pure NumPy ───────────────────────────────────
class TinyMLP:
    """2-hidden-layer MLP with tanh activation for PINN demo."""
    def __init__(self, layer_sizes, seed=42):
        rng = np.random.RandomState(seed)
        self.weights = []
        self.biases = []
        for i in range(len(layer_sizes) - 1):
            fan_in, fan_out = layer_sizes[i], layer_sizes[i+1]
            W = rng.randn(fan_in, fan_out) * np.sqrt(2.0 / (fan_in + fan_out))
            b = np.zeros(fan_out)
            self.weights.append(W)
            self.biases.append(b)

    def forward(self, x):
        """Forward pass returning u, du/dx, d2u/dx2 via manual chain rule."""
        # x is shape (N, 1)
        # Layer 1: z1 = W1*x + b1, a1 = tanh(z1)
        z1 = x @ self.weights[0] + self.biases[0]
        a1 = np.tanh(z1)
        da1 = 1.0 - a1**2          # tanh'(z1)
        dda1 = -2.0 * a1 * da1     # tanh''(z1)

        # Layer 2: z2 = W2*a1 + b2, a2 = tanh(z2)
        z2 = a1 @ self.weights[1] + self.biases[1]
        a2 = np.tanh(z2)
        da2 = 1.0 - a2**2
        dda2 = -2.0 * a2 * da2

        # Output: u = W3*a2 + b3  (linear output)
        u = a2 @ self.weights[2] + self.biases[2]  # (N, 1)

        # du/dx via chain rule:
        # du/da2 = W3^T, da2/dz2 = da2, dz2/da1 = W2^T, da1/dz1 = da1, dz1/dx = W1^T
        # du/dx = W3 * da2 * W2 * da1 * W1  (element-wise along paths)
        # For scalar output, this simplifies:
        # du/dz2 = W3^T shape (hidden,)
        du_dz2 = da2 * (self.weights[2].T)  # (N, hidden2)
        du_da1 = du_dz2 @ self.weights[1].T  # (N, hidden1)
        du_dz1 = du_da1 * da1  # (N, hidden1)
        dudx = du_dz1 @ self.weights[0].T  # (N, 1)

        # d2u/dx2 (second derivative through the chain rule)
        # Need to differentiate dudx w.r.t. x again
        # d(du_dz2)/dx = dda2 * W3^T * (dz2/dx)
        # dz2/dx = W2^T * da1 * W1^T
        dz2_dx = (da1 @ self.weights[1]) * 1.0  # we need dz2/dx per sample
        # Actually let's compute d2u/dx2 numerically for simplicity
        # (manual second derivative is complex; use finite differences on du/dx)
        # This is the toy demo; production code uses autograd.
        return u, dudx

    def forward_with_d2(self, x, eps=1e-5):
        """Forward pass with second derivative via finite differences on dudx."""
        u, dudx = self.forward(x)
        _, dudx_plus = self.forward(x + eps)
        _, dudx_minus = self.forward(x - eps)
        d2udx2 = (dudx_plus - dudx_minus) / (2 * eps)
        return u, dudx, d2udx2

    def get_params(self):
        return [w.copy() for w in self.weights] + [b.copy() for b in self.biases]

    def set_params(self, params):
        n = len(self.weights)
        for i in range(n):
            self.weights[i] = params[i].copy()
            self.biases[i] = params[i + n].copy()

# ── PINN Training ────────────────────────────────────────────
net = TinyMLP([1, 20, 20, 1])

# Collocation points
N_col = 100
x_col = np.linspace(0, 2*np.pi, N_col).reshape(-1, 1)

# Training parameters
lr = 0.001
lambda_ic = 50.0
n_epochs = 3000
losses = []

for epoch in range(n_epochs):
    # Forward pass at collocation points
    u, dudx, d2udx2 = net.forward_with_d2(x_col)

    # PDE residual: y'' + y = 0
    residual = d2udx2 + u
    loss_pde = np.mean(residual**2)

    # IC: y(0) = 0, y'(0) = 1
    u0, dudx0, _ = net.forward_with_d2(np.array([[0.0]]))
    loss_ic = (u0[0,0] - 0.0)**2 + (dudx0[0,0] - 1.0)**2

    loss = loss_pde + lambda_ic * loss_ic
    losses.append(loss)

    # Numerical gradient update (finite differences on each parameter)
    eps_grad = 1e-5
    params = net.get_params()
    grads = []
    for i, p in enumerate(params):
        g = np.zeros_like(p)
        for idx in np.ndindex(p.shape):
            params_plus = [pp.copy() for pp in params]
            params_plus[i][idx] += eps_grad
            net.set_params(params_plus)
            u_p, _, d2_p = net.forward_with_d2(x_col)
            r_p = d2_p + u_p
            l_pde_p = np.mean(r_p**2)
            u0_p, du0_p, _ = net.forward_with_d2(np.array([[0.0]]))
            l_ic_p = (u0_p[0,0])**2 + (du0_p[0,0] - 1.0)**2
            loss_plus = l_pde_p + lambda_ic * l_ic_p

            params_minus = [pp.copy() for pp in params]
            params_minus[i][idx] -= eps_grad
            net.set_params(params_minus)
            u_m, _, d2_m = net.forward_with_d2(x_col)
            r_m = d2_m + u_m
            l_pde_m = np.mean(r_m**2)
            u0_m, du0_m, _ = net.forward_with_d2(np.array([[0.0]]))
            l_ic_m = (u0_m[0,0])**2 + (du0_m[0,0] - 1.0)**2
            loss_minus = l_pde_m + lambda_ic * l_ic_m

            g[idx] = (loss_plus - loss_minus) / (2 * eps_grad)
        grads.append(g)

    # Gradient descent
    for i in range(len(params)):
        params[i] -= lr * grads[i]
    net.set_params(params)

    if (epoch + 1) % 500 == 0:
        print(f"Epoch {epoch+1}: loss={loss:.6f}, PDE={loss_pde:.6f}, IC={loss_ic:.6f}")

# ── Evaluation ───────────────────────────────────────────────
x_test = np.linspace(0, 2*np.pi, 200).reshape(-1, 1)
u_pred, _, _ = net.forward_with_d2(x_test)
u_exact = np.sin(x_test)

fig, axes = plt.subplots(1, 2, figsize=(12, 4))
axes[0].plot(x_test, u_exact, 'b-', lw=2, label='Exact: sin(x)')
axes[0].plot(x_test, u_pred, 'r--', lw=2, label='PINN prediction')
axes[0].set_xlabel('x'); axes[0].set_ylabel('y')
axes[0].legend(); axes[0].set_title('PINN Solution for y\'\' + y = 0')
axes[0].grid(True, alpha=0.3)

axes[1].semilogy(losses)
axes[1].set_xlabel('Epoch'); axes[1].set_ylabel('Total Loss')
axes[1].set_title('Training History')
axes[1].grid(True, alpha=0.3)
plt.tight_layout()
plt.show()

print(f"Max absolute error: {np.max(np.abs(u_pred - u_exact)):.4f}")

Agent Lens: The PINN as a Self-Supervised Agent

Adaptive collocation as active learning. A key design choice in PINNs is where to place collocation points. Uniform random sampling wastes evaluations in smooth regions. A smarter agent samples more collocation points where the PDE residual is large—this is residual-based adaptive refinement (RAR). At each training stage, the agent evaluates the residual on a coarse grid, identifies high-residual regions, and adds new collocation points there. This is directly analogous to active learning: the agent chooses the most informative training examples.

Loss balancing as reward shaping. The weights $\lambda_{\text{PDE}}, \lambda_{\text{BC}}, \lambda_{\text{IC}}$ shape the effective reward landscape. Wang et al. (2021) proposed using the Neural Tangent Kernel (NTK) to set weights that balance the learning rates of different loss components. This is analogous to reward shaping in RL: the raw reward (PDE residual) is supplemented with auxiliary rewards (BC, IC satisfaction) to guide learning.

Curriculum learning. Some problems benefit from a training curriculum: first train with low-frequency collocation (or on an easier sub-domain), then gradually increase difficulty. This parallels curriculum learning in RL, where agents first learn simple tasks before tackling hard ones.

Exercises

Analytical

Exercise 1. Write out the PINN loss function for the advection equation $u_t + cu_x = 0$ on $[0,1]\times[0,1]$ with $u(x,0) = \sin(2\pi x)$ and periodic boundary conditions.

Exercise 2. For a network $u_\theta(x) = w_2\tanh(w_1 x + b_1) + b_2$ with scalar parameters, compute $du_\theta/dx$ and $d^2u_\theta/dx^2$ explicitly in terms of $w_1, w_2, b_1$.

Exercise 3. Explain why the total PINN error has three components (approximation, optimization, generalization) and give a concrete example of when each dominates.

Exercise 4. Show that for the 1D Poisson equation $-u'' = f(x)$, $u(0) = u(1) = 0$, the PINN residual loss is $\mathcal{L}_{\text{PDE}} = \frac{1}{N}\sum |u''_\theta(x_i) + f(x_i)|^2$.

Computational

Exercise 5. Modify the toy PINN code to solve $y'' + 4y = 0$, $y(0) = 1$, $y'(0) = 0$ (solution: $\cos(2x)$). Report the maximum error on $[0, \pi]$.

Exercise 6. Implement a PINN for the exponential decay ODE $y' + 2y = 0$, $y(0) = 3$ (solution: $3e^{-2x}$). Since this is first-order, you only need $u$ and $u'$.

Exercise 7. Add a data-fitting term to the code lab: generate 20 noisy observations of $\sin(x)$ and include them in the loss. Compare the result with and without the data term.

Exercise 8. Implement residual-based adaptive refinement: every 500 epochs, evaluate the PDE residual on a fine grid, identify the 10 points with largest residual, and add them to the collocation set. Does this improve accuracy?

Agentic

Exercise 9. Design an agent that adaptively adjusts $\lambda_{\text{IC}}$ during training. Start with $\lambda_{\text{IC}} = 100$ and reduce it as the IC loss decreases below a threshold. Compare the final accuracy with fixed $\lambda_{\text{IC}}$.

Exercise 10. Implement a “multi-task PINN agent” that trains one network to solve $y'' + ky = 0$ for $k = 1, 4, 9$ simultaneously (with $k$ as an additional input). How does the agent balance accuracy across the three tasks?

Assessment

Quiz

Mini-Project: PINN for the Diffusion Equation

Solve $u_t = 0.01\,u_{xx}$ on $[0,1] \times [0,1]$ with $u(x,0) = \sin(\pi x)$ and $u(0,t) = u(1,t) = 0$.

Deliverables:

1. Implement a NumPy PINN (or use the PyTorch notebook as a starting point)
2. Plot the learned solution as a heatmap over $(x, t)$
3. Compare with the exact solution $u(x,t) = e^{-0.01\pi^2 t}\sin(\pi x)$
4. Implement a finite-difference solver and compare accuracy and runtime
5. Report the relative $L^2$ error for both methods

References & Next Steps

References

1. Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2019). “Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations.” Journal of Computational Physics, 378, 686–707.
2. Lagaris, I. E., Likas, A., & Fotiadis, D. I. (1998). “Artificial neural networks for solving ordinary and partial differential equations.” IEEE Transactions on Neural Networks, 9(5), 987–1000.
3. Lu, L., Meng, X., Mao, Z., & Karniadakis, G. E. (2021). “DeepXDE: A deep learning library for solving differential equations.” SIAM Review, 63(1), 208–228.
4. Karniadakis, G. E., Kevrekidis, I. G., Lu, L., Perdikaris, P., Wang, S., & Yang, L. (2021). “Physics-informed machine learning.” Nature Reviews Physics, 3, 422–440.
5. Wang, S., Teng, Y., & Perdikaris, P. (2021). “Understanding and mitigating gradient flow pathologies in physics-informed neural networks.” SIAM Journal on Scientific Computing, 43(5), A3055–A3081.
6. Shin, Y., Darbon, J., & Karniadakis, G. E. (2020). “On the convergence of physics informed neural networks.” arXiv:2004.01806.

Full PyTorch Notebook

For a complete PINN implementation with automatic differentiation solving the Burgers equation (forward and inverse): notebooks/pinn_full.ipynb

Next Steps

• Module 3.3: Neural Operators — learn entire solution operators instead of individual solutions.
• Module 3.1 ← revisit Neural ODEs for comparison with the PINN approach.