maximuspowers/sig-autoencoder-fourier-5-supervised-mse

Weight-Space Autoencoder (TRANSFORMER)

This model is a weight-space autoencoder trained on neural network activation weights/signatures. It includes both an encoder (compresses weights into latent representations) and a decoder (reconstructs weights from latent codes).

Model Description

• Architecture: Transformer encoder-decoder
• Training Dataset: maximuspowers/muat-fourier-5
• Input Mode: signature
• Latent Dimension: 128

Tokenization

• Chunk Size: 1 weight values per token
• Max Tokens: 64
• Metadata: True

Training Config

• Loss Function: contrastive
• Optimizer: adamw
• Learning Rate: 0.0001
• Batch Size: 32

Performance Metrics (Test Set)

• MSE: 0.088750
• MAE: 0.187620
• RMSE: 0.297909
• Cosine Similarity: 0.9603
• R² Score: 0.9855