Rename model notation reference (#13)

MODEL_NOTATION.md

@@ -0,0 +1,206 @@
+# Formal Model Specifications
+
+This document formalises the statistical models implemented in
+`src/ners/research/models`. Throughout, the training set is
+$\mathcal{D} = \{(\mathbf{x}^{(i)}, y^{(i)})\}_{i=1}^N$ with labels
+$y^{(i)} \in \{0,1\}$ for the binary gender classes. Feature vectors
+$\mathbf{x}^{(i)}$ combine
+
+* character $n$-gram count representations of name strings produced by
+  `CountVectorizer` or `TfidfVectorizer`, and
+* engineered scalar or categorical metadata (e.g., name length, province)
+  that are either used directly or encoded by `LabelEncoder`.
+
+For neural architectures, character or token sequences are converted into
+integer index sequences using a `Tokenizer` before being padded to a
+maximum length specified in the configuration. Predictions are returned as
+class posterior probabilities via a softmax layer unless otherwise noted.
+
+## Logistic Regression (`logistic_regression_model.py`)
+
+**Feature map.** Character $n$-gram counts
+$\phi(\mathbf{x}) \in \mathbb{R}^d$ obtained with
+`CountVectorizer(analyzer="char", ngram_range=(2,4))` (default configuration).【F:src/ners/research/models/logistic_regression_model.py†L16-L46】
+
+**Model.** The linear logit for class $1$ is
+$z = \mathbf{w}^\top \phi(\mathbf{x}) + b$. The class posteriors are
+$p(y=1\mid \mathbf{x}) = \sigma(z)$ and $p(y=0\mid \mathbf{x}) = 1 - \sigma(z)$
+with $\sigma(u) = (1 + e^{-u})^{-1}$.
+
+**Training objective.** Minimise the regularised negative log-likelihood
+
+$$\mathcal{L}(\mathbf{w}, b) = -\sum_{i=1}^N \left[y^{(i)}\log p(y^{(i)}=1\mid \mathbf{x}^{(i)}) + (1-y^{(i)}) \log p(y^{(i)}=0\mid \mathbf{x}^{(i)})\right] + \lambda R(\mathbf{w}),$$
+
+where $R$ is the penalty induced by the chosen solver (e.g., $\ell_2$ for
+`liblinear`).
+
+## Multinomial Naive Bayes (`naive_bayes_model.py`)
+
+**Feature map.** Character $n$-gram counts
+$\phi(\mathbf{x}) \in \mathbb{N}^d$ derived with
+`CountVectorizer(analyzer="char", ngram_range=(2,4))` by default.【F:src/ners/research/models/naive_bayes_model.py†L15-L38】
+
+**Generative model.** For each class $c \in \{0,1\}$, the class prior is
+$\pi_c = \frac{N_c}{N}$. Conditional feature probabilities are estimated with
+Laplace smoothing (parameter $\alpha$):
+
+$$\theta_{cj} = \frac{N_{cj} + \alpha}{\sum_{k=1}^d (N_{ck} + \alpha)},$$
+
+where $N_{cj}$ counts the total occurrences of feature $j$ among examples of
+class $c$. The likelihood of an input with counts $\phi_j(\mathbf{x})$ is
+
+$$p(\phi(\mathbf{x})\mid y=c) = \prod_{j=1}^d \theta_{cj}^{\phi_j(\mathbf{x})}.$$
+
+**Inference.** Predict with the maximum a posteriori (MAP) decision
+$\hat{y} = \arg\max_c \left\{ \log \pi_c + \sum_{j=1}^d \phi_j(\mathbf{x}) \log \theta_{cj} \right\}$.
+
+## Random Forest (`random_forest_model.py`)
+
+**Feature map.** Concatenation of engineered numerical features and label
+encoded categorical attributes produced on demand in
+`prepare_features`.【F:src/ners/research/models/random_forest_model.py†L28-L71】
+
+**Model.** An ensemble of $T$ decision trees ${\{T_t\}}_{t=1}^T$, each trained on
+a bootstrap sample of the data with random feature sub-sampling. Each tree
+outputs a class prediction $T_t(\mathbf{x}) \in \{0,1\}$. The forest prediction
+is the mode of individual votes:
+
+$$\hat{y} = \operatorname{mode}\{T_t(\mathbf{x}) : t = 1,\dots,T\}.$$ 
+
+**Class probability.** For soft outputs,
+$p(y=1\mid \mathbf{x}) = \frac{1}{T} \sum_{t=1}^T p_t(y=1\mid \mathbf{x})$ where
+$p_t$ is the class distribution estimated at the leaf reached by
+$\mathbf{x}$ in tree $t$.
+
+## LightGBM (`lightgbm_model.py`)
+
+**Feature map.** Hybrid of numeric inputs, categorical label encodings, and
+character $n$-gram counts expanded into dense columns and assembled into a
+feature matrix persisted in `self.feature_columns`.【F:src/ners/research/models/lightgbm_model.py†L38-L118】
+
+**Model.** Gradient boosted decision trees forming an additive function
+$F_M(\mathbf{x}) = \sum_{m=0}^M \eta h_m(\mathbf{x})$, where $h_m$ denotes the
+$m$-th tree and $\eta$ is the learning rate.
+
+**Training objective.** LightGBM minimises
+
+$$\mathcal{L}^{(m)} = \sum_{i=1}^N \ell\big(y^{(i)}, F_{m-1}(\mathbf{x}^{(i)}) + h_m(\mathbf{x}^{(i)})\big) + \Omega(h_m),$$
+
+using second-order Taylor approximations of the loss $\ell$ (binary log-loss by
+default) and regulariser $\Omega$ determined by tree complexity constraints.
+
+## XGBoost (`xgboost_model.py`)
+
+**Feature map.** Combination of numeric metadata, categorical label encodings,
+and character $n$-gram counts as described in `prepare_features`.【F:src/ners/research/models/xgboost_model.py†L41-L113】
+
+**Model.** Additive ensemble of regression trees
+$F_M(\mathbf{x}) = \sum_{m=1}^M f_m(\mathbf{x})$ with $f_m \in \mathcal{F}$, the
+space of trees with fixed structure.
+
+**Training objective.** At boosting iteration $m$, minimise the regularised
+objective
+
+$$\mathcal{L}^{(m)} = \sum_{i=1}^N \ell\big(y^{(i)}, F_{m-1}(\mathbf{x}^{(i)}) + f_m(\mathbf{x}^{(i)})\big) + \Omega(f_m),$$
+
+where $\Omega(f) = \gamma T_f + \tfrac{1}{2} \lambda \sum_{j=1}^{T_f} w_j^2$ penalises the
+number of leaves $T_f$ and their scores $w_j$. The optimal leaf weights follow
+
+$$w_j^{\star} = - \frac{\sum_{i \in I_j} g_i}{\sum_{i \in I_j} h_i + \lambda},$$
+
+with $g_i$ and $h_i$ denoting first- and second-order gradients of the loss for
+sample $i$.
+
+## Convolutional Neural Network (`cnn_model.py`)
+
+**Input encoding.** Character-level token sequences padded to length
+$L$ using `Tokenizer(char_level=True)` followed by `pad_sequences`.【F:src/ners/research/models/cnn_model.py†L23-L64】
+
+**Architecture.** Embedding layer producing $X \in \mathbb{R}^{L \times d}$,
+followed by two convolutional blocks:
+
+1. $H^{(1)} = \operatorname{ReLU}(\operatorname{Conv1D}_{k_1}(X))$ with kernel size
+   $k_1$ and $F$ filters, then temporal max-pooling.
+2. $H^{(2)} = \operatorname{ReLU}(\operatorname{Conv1D}_{k_2}(H^{(1)}))$ with kernel
+   size $k_2$ and $F$ filters.
+
+Global max-pooling yields $h = \max_{t} H^{(2)}_{t,:}$, which passes through a
+dense layer and dropout before the softmax layer producing
+$p(y\mid \mathbf{x}) = \operatorname{softmax}(W h + b)$.
+
+**Loss.** Cross-entropy between softmax output and the ground-truth label.
+
+## Bidirectional GRU (`bigru_model.py`)
+
+**Input encoding.** Word-level sequences padded to length $L$ with
+`Tokenizer(char_level=False)` and `pad_sequences`.【F:src/ners/research/models/bigru_model.py†L47-L69】
+
+**Recurrent dynamics.** A stacked bidirectional GRU computes forward and
+backward hidden states according to
+
+\[
+\begin{aligned}
+\mathbf{z}_t &= \sigma(W_z \mathbf{x}_t + U_z \mathbf{h}_{t-1} + \mathbf{b}_z),\\
+\mathbf{r}_t &= \sigma(W_r \mathbf{x}_t + U_r \mathbf{h}_{t-1} + \mathbf{b}_r),\\
+\tilde{\mathbf{h}}_t &= \tanh\big(W_h \mathbf{x}_t + U_h(\mathbf{r}_t \odot \mathbf{h}_{t-1}) + \mathbf{b}_h\big),\\
+\mathbf{h}_t &= (1 - \mathbf{z}_t) \odot \mathbf{h}_{t-1} + \mathbf{z}_t \odot \tilde{\mathbf{h}}_t.
+\end{aligned}
+\]
+
+The final representation concatenates the last forward and backward states
+before passing through dense layers and a softmax classifier.
+
+## Bidirectional LSTM (`lstm_model.py`)
+
+**Input encoding.** Word-level sequences padded to length $L$ using the same
+pipeline as the BiGRU model.【F:src/ners/research/models/lstm_model.py†L45-L67】
+
+**Recurrent dynamics.** At each timestep, the LSTM updates its memory cell via
+
+\[
+\begin{aligned}
+\mathbf{i}_t &= \sigma(W_i \mathbf{x}_t + U_i \mathbf{h}_{t-1} + \mathbf{b}_i),\\
+\mathbf{f}_t &= \sigma(W_f \mathbf{x}_t + U_f \mathbf{h}_{t-1} + \mathbf{b}_f),\\
+\mathbf{o}_t &= \sigma(W_o \mathbf{x}_t + U_o \mathbf{h}_{t-1} + \mathbf{b}_o),\\
+\tilde{\mathbf{c}}_t &= \tanh(W_c \mathbf{x}_t + U_c \mathbf{h}_{t-1} + \mathbf{b}_c),\\
+\mathbf{c}_t &= \mathbf{f}_t \odot \mathbf{c}_{t-1} + \mathbf{i}_t \odot \tilde{\mathbf{c}}_t,\\
+\mathbf{h}_t &= \mathbf{o}_t \odot \tanh(\mathbf{c}_t).
+\end{aligned}
+\]
+
+Bidirectional aggregation concatenates terminal forward/backward hidden vectors
+before the dense-softmax head.
+
+## Transformer Encoder (`transformer_model.py`)
+
+**Input encoding.** Token sequences padded to a fixed length with positional
+indices $\{0, \ldots, L-1\}$ added through a learned positional embedding.
+`Tokenizer` initialises the vocabulary; padding uses `pad_sequences`.【F:src/ners/research/models/transformer_model.py†L25-L77】
+
+**Architecture.** For hidden dimension $d$, the encoder block computes
+
+\[
+\begin{aligned}
+Z^{(0)} &= X + P,\\
+Z^{(1)} &= \operatorname{LayerNorm}\big(Z^{(0)} + \operatorname{Dropout}(\operatorname{MHAttn}(Z^{(0)}))\big),\\
+Z^{(2)} &= \operatorname{LayerNorm}\big(Z^{(1)} + \operatorname{Dropout}(\phi(Z^{(1)} W_1 + b_1) W_2 + b_2)\big),
+\end{aligned}
+\]
+
+where $\operatorname{MHAttn}$ is multi-head self-attention with
+$H$ heads. Global average pooling produces a fixed-length vector for the dense
+and dropout layers before the final softmax classifier.
+
+## Ensemble Voting (`ensemble_model.py`)
+
+**Base learners.** A configurable set of pipelines that include character
+$n$-gram vectorisers and classical classifiers (logistic regression,
+random forest, naive Bayes).【F:src/ners/research/models/ensemble_model.py†L29-L96】
+
+**Aggregation.** Given model posteriors $\mathbf{p}_j(\mathbf{x})$ and non-negative
+weights $w_j$, the soft-voting ensemble predicts
+
+$$p(y=c\mid \mathbf{x}) = \frac{1}{\sum_j w_j} \sum_j w_j \, p_j(y=c\mid \mathbf{x}).$$
+
+Hard voting instead returns
+$\hat{y} = \operatorname{mode}\{\arg\max_c p_j(y=c\mid \mathbf{x})\}$.