README.md

---
title: TuRTLe Leaderboard
emoji: 🐢
colorFrom: gray
colorTo: green
sdk: gradio
app_file: app.py
pinned: true
license: apache-2.0
short_description: A Unified Evaluation of LLMs for RTL Generation.
sdk_version: 5.39.0
---

## Quick Introduction

### Prerequisites

- **Python 3.11** or higher (required by the project)
- **[uv](https://docs.astral.sh/uv/getting-started/installation/)** for managing dependencies (optional but recommended)

#### Installing uv (optional)

On macOS and Linux:
```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```

Install dependencies:
```bash
uv sync

# or using regular python
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```

### Deploy Locally

```
$ uv run app.py # or python3 app.py
* Running on local URL:  http://127.0.0.1:7860
* To create a public link, set `share=True` in `launch()`.
```

Then on localhost http://127.0.0.1:7860 you should have the leaderboard running

### Add new models

If you are from outside of HPAI you must directly modify the `results/results_icarus.json` and `results/results_verilator.json` files.

If you are from HPAI, you can add your model onto our shared `.csv` file of results and follow these steps:

1. Modify the `config/model_metadata.py` file, `MODELS` dictionary to include a new entry for your model

    For example, if we wish to include the classic GPT2 model, we would add the following metadata:

    ```python
    MODELS = {
        ...

        "GPT2": ModelMetadata(
            "https://huggingface.co/openai-community/gpt2",  # model url
            0.13,  # params (in B)
            "Coding",  # model type: "General", "Coding", or "RTL-Specific"
            "V1",  # release of the TuRTLe Leaderboard: "V1", "V2", or "V3"
            "Dense"  # model architecture: "Dense" or "Reasoning"
        ),
    }
    ```

2. Parse the CSV files onto JSON, which is what the Leaderboard will take as ground truth

    ```
    $ uv run -m results.parse results/results_v3_mlcad_icarus.csv # will generate results/results_v3_mlcad_icarus.json
    $ uv run -m results.parse results/results_v3_mlcad_verilator.csv # will generate results/results_v3_mlcad_verilator.json
    ```

    The application is hardcoded to look for `results_icarus.json` and `results_verilator.json.` Rename the files you just created:

    ```
    $ mv results/results_v3_mlcad_icarus.json results/results_icarus.json
    $ mv results/results_v3_mlcad_verilator.json results/results_verilator.json
    ```

3. Compute the aggregated scores

    This will generate the corresponding `aggregated_scores` files that the leaderboard uses for some of its views.

    ```
    $ uv run results/compute_agg_results.py results/results_v3_mlcad_icarus.csv
    $ uv run results/compute_agg_results.py results/results_v3_mlcad_verilator.csv
    ```

    This will create `aggregated_scores_v3_mlcad_icarus.csv` and `aggregated_scores_v3_mlcad_verilator.csv`. Rename them to what the application expects:

    ```
    $ mv results/aggregated_scores_v3_mlcad_icarus.csv results/aggregated_scores_icarus.csv
    $ mv results/aggregated_scores_v3_mlcad_verilator.csv results/aggregated_scores_verilator.csv
    ```


## License

This project is licensed under the Apache License 2.0.  
See the [LICENSE](./LICENSE) and [NOTICE](./NOTICE) files for more details.