Architecture
NawalTransformer is a 100% sovereign language model built from scratch. Three model sizes serve different deployment scenarios, from edge devices to full-scale inference nodes. A genome evolution system encodes and optimizes architectures as compact DNA sequences, enabling evolutionary search across neural topologies.
Architecture Details
NawalTransformer
Pre-norm transformer blocks with LayerNorm → MultiHeadAttention → residual → LayerNorm → FFN → residual. GELU activation, weight tying between embeddings and LM head, and KV caching for generation.
Three Model Sizes
nawal_small (117M: 768 hidden, 12 layers, 12 heads), nawal_medium (350M: 1024 hidden, 24 layers, 16 heads), nawal_large (1.3B: 1536 hidden, 36 layers, 24 heads).
Genome Evolution System
DNA encoding compresses neural architecture to ~5KB vs 500MB weights. 30+ LayerTypes including MoE, SSM, Flash Attention, and RoPE with mutation, crossover, and selection operators.
GenomeToNawalAdapter
Pipeline from Genome DNA → GenomeEncoder → NawalConfig → NawalTransformer. ModelBuilder supports all 30+ layer types with FLOPs and memory estimation.
Fitness Evaluation
Three-factor scoring: quality (40%), timeliness (30%), honesty (30%). PopulationManager tracks genome populations across evolutionary generations.
Multilingual Embeddings
NawalEmbeddings with token + positional encoding, Xavier/Glorot initialization, and a 52,000 token vocabulary supporting 5 Belizean languages: English, Spanish, Kriol, Garifuna, and Mopan Maya.