FoSSIL: A Unified Framework for Continual Semantic Segmentation in 2D and 3D Domains

Continual semantic segmentation remains an underexplored area in both 2D and 3D domains. The problem becomes particularly challenging when classes and domains evolve over time, with incremental classes having only a few labeled samples. In this setting, the model must simultaneously address catastrophic forgetting of old classes, overfitting due to the limited labeled data of new classes, and domain shifts arising from changes in data distribution. Existing methods fail to simultaneously address these real-world constraints. We introduce the FoSSIL framework, which integrates guided noise injection and prototype-guided pseudo-label refinement (PLR) to enhance continual learning across class-incremental (CIL), domain-incremental (DIL), and few-shot scenarios. Guided noise injection perturbs parameters with overfitted or saturated gradients more strongly, while perturbing parameters with highly changing or large gradients less, preserving critical weights, allowing less critical parameters to explore alternative solutions in the parameter space, mitigating forgetting, reducing overfitting, and improving robustness to domain shifts. For incremental classes with unlabeled data, PLR enables semi-supervised learning by refining pseudo-labels and filtering out incorrect high-confidence predictions, ensuring reliable supervision for incremental classes. Together, these components work synergistically to enhance stability, generalization, and continual learning across all learning regimes.

Overview

FoSSIL Framework

Problem Definition

We formalize the multi-constrained continual learning problem for semantic segmentation as a sequence of sessions \( \mathcal{S} = \{\mathcal{S}_0, \mathcal{S}_1, \dots, \mathcal{S}_T\} \) where each session \( \mathcal{S}_t \) has a semantic class space \( \mathcal{C}_t \) and domain distribution \( \mathcal{D}_t \). Each session has a dataset \( \mathbb{D}_t = \{(x_i, y_i)\}_{i=1}^{N_t} \) with input images \( x_i \in \mathcal{X} \) and pixel-wise labels \( y_i \in \mathcal{Y}_t \subseteq \mathcal{C}_t \).

Classes and Domains across Sessions

• Same Classes, Different Domains: \( \mathcal{C}_t = \mathcal{C}_{t'} \), \( \mathcal{D}_t \neq \mathcal{D}_{t'} \)
• Different Classes, Same Domain: \( \mathcal{C}_t \neq \mathcal{C}_{t'} \), \( \mathcal{D}_t = \mathcal{D}_{t'} \)
• Different Classes, Different Domain: \( \mathcal{C}_t \neq \mathcal{C}_{t'} \), \( \mathcal{D}_t \neq \mathcal{D}_{t'} \)

Few-shot Learning

Incremental sessions (\( t \neq 0 \)) have few-shot labeled data \( N_t \ll N_0 \), with \( N_t = K \cdot |\mathcal{C}_t| \), where \( K \in \{5,10,20,30\} \). Each session may also access unlabeled data \( \mathcal{U}_t = \{x_j^{(u)}\}_{j=1}^{M_t} \), where \( M_t \gg N_t \).

Exemplar-Free Prototype Replay

For each class \( c \in \mathcal{C}_t \), a compact prototype is extracted from the feature space using a model feature extractor \( \phi \). For each sample \( (x_i, y_i) \), the embedding is \( E_i = \phi(x_i) \). The class prototype is computed as:

\( p_c^{(i)} = \frac{1}{|\mathcal{F}_c^{(i)}|} \sum_{\mathbf{f} \in \mathcal{F}_c^{(i)}} \mathbf{f} \)

\( P_c = \frac{1}{NS_c} \sum_{j \in NS_c} \frac{p_c^{(j)}}{\|p_c^{(j)}\|_2} \)

The prototype replay loss is:

\( \mathcal{L}_{\text{proto}} = \sum_{P_c \in \mathcal{P}_t} \mathcal{L}_{\text{CE}}(F(P_c), c) \)

Guided Noise Injection

Weights are perturbed according to gradient-based guidance:

\( G_{ij}^{-1} = \frac{1}{G_{ij} + \epsilon} \)

\( \tilde{G}_{ij}^{-1} = \frac{1 + G_{ij}^{-1} - \min(G^{-1})}{1 + \max(G^{-1}) - \min(G^{-1})} \)

\( \tilde{\mathbb{W}} = \mathbb{W} + \tilde{G}^{-1} \odot \mathcal{N}(0, I) \)

Prototype-Guided Pseudo-Label Refinement (PLR)

PLR uses class prototypes to refine pseudo-labels for unlabeled data \( x_j^{(u)} \), ensuring reliable supervision for incremental classes. Student and teacher networks generate pseudo-labels:

\( \hat{y}_s, \mathcal{F}_s = M_s(x_j^{(u)}), \quad \hat{y}_t, \mathcal{F}_t = M_t(x_j^{(u)}) \)

Pseudo-labels are retained only if confidence and feature similarity with prototypes meet thresholds:

\( \text{valid}(p,q) = (\text{conf}(p,q) > \tau_{\text{conf}}) \land (\text{sim}(p,q) > \tau_{\text{sim}}) \)

Consistency loss on validated pixels:

\( \mathcal{L}_{\text{consistency}} = \frac{1}{|\mathcal{V}|} \sum_{(p,q) \in \mathcal{V}} \|\hat{y}_s(p,q) - \hat{y}_t(p,q)\|_2^2 \)

Dataset Details

FoSSIL Datasets

Incremental Sessions

Results

Performance of baselines on Med FoSSIL-Disjoint benchmark (3 sessions)

Results are reported as Dice coefficients (0–1).
PD (Performance Drop rate) = ((Session 0 − Session 2) / Session 0) × 100.

Performance on Natural-FoSSIL benchmark

All values are reported as mIoU (0–100).
PD (Performance Drop rate) = ((Session 0 − Session 2) / Session 0) × 100.

Performance of recent prototype replay-based and semi-supervised methods

Results reported as Dice coefficients (0–1).

Performance of FoSSIL on the Med FoSSIL-Disjoint benchmark and its variant, Med Semi-Supervised-FoSSIL (which includes additional unlabeled data), evaluated across incremental sessions (TS (Base) → AMOS → BCV → MOTS → BraTS → VerSe).

Seen refers to the average performance on classes the model has encountered in previous sessions, while New refers to the average performance on classes introduced in the current session. Results are reported as Dice coefficients (0–1). The reported results confirm that unlabeled data helps to boost the performance.

SS refers to Semi-Supervised, and HM refers to Harmonic Mean. SS FoSSIL denotes the performance of FoSSIL on the Med Semi-Supervised-FoSSIL benchmark.

Performance of FoSSIL on the Semi-Supervised Natural-FoSSIL benchmark (which includes additional unlabeled data), evaluated across incremental sessions (BDD100K (Base) → Cityscapes → IDD → IDD (with different classes from the previous session)).

Seen refers to the average performance on classes the model has encountered in previous sessions, while New refers to the average performance on classes introduced in the current session. Results are reported as mIoU (0–100). The results show that pseudo-label refinement (PRL) enhances the performance of FoSSIL compared to FoSSIL without pseudo-label refinement (w/o PRL).

HM refers to Harmonic Mean.

Performance of FoSSIL (MedFormer) on the Med FoSSIL-Mixed benchmark, evaluated across incremental sessions (Session 0 (Base) → Session 1 → Session 2 → Session 3 → Session 4).

Seen refers to the average performance on classes the model has encountered in previous sessions, while New refers to the average performance on classes introduced in the current session. Results are reported as Dice coefficients (0–1). The results show that guided noise injection (GNI) enhances the performance of FoSSIL compared to FoSSIL without guided noise injection (w/o GNI).

HM refers to Harmonic Mean.

Baselines

Class-Incremental Semantic Segmentation

• MiB: A class-incremental learning method for semantic segmentation in natural images.
• CLIP-CT: A continual learning method for abdominal multi-organ and tumour segmentation.
• Saving100x: Leverages prototype replay for incremental semantic segmentation.
• Adaptive Prototype: Updates prototypes adaptively for class-incremental learning.

These methods cannot handle few-shot learning and domain shifts.

Domain-Incremental Learning

• MDIL: Uses multiple decoders for different domains with a shared encoder.

Separate decoder per domain limits scalability; cannot handle few-shot incremental classes.

Few-shot Class-Incremental Learning

• PIFS: Prototype-based few-shot CIL.
• Subspace: Regularization-based few-shot CIL.
• C-FSCIL: Meta-learning for few-shot CIL.
• FACT: Reserves embedding space for new classes.
• NC-FSCIL: Fixes classifier as simplex ETF.
• Gen-Replay: Data-free replay using diffusion model for 3D volumes.
• GAPS: Guided copy-paste augmentation.
• SoftNet: Learns model weights & adaptive soft masks.
• FSCIL-SS: Pseudo-labeling & knowledge distillation.
• FeCAM: Enhances class prototype representation; evaluates domain-incremental learning separately.

Although effective for class-incremental and few-shot learning, these methods are not robust to domain shifts.

Semi-Supervised Based Methods

• RETRIEVE: Robust semi-supervised learning via coreset selection.
• NNCSL: Nearest-neighbor based continual semi-supervised learning.
• UaD-CE: Class Equilibrium + Uncertainty-aware Distillation.
• CSL: Pseudo-label selection as convex optimization.

Cannot jointly address domain shifts, class-incremental, and few-shot learning.

Others (Representation Learning, Few-shot Learning, Meta-Learning, Active Learning, Domain-shift)

• SupCL, UnSupCL, UnSupCL-HNM: Contrastive learning methods.
• MTL: Fast adaptation to unseen tasks; few-shot learning.
• MAML: Multi-task representation learning; few-shot learning.
• CLIP-driven: Language-Vision model for organ segmentation & tumor detection.
• HALO: Hyperbolic NN for pixel-level active learning under domain shift.