Guides, benchmarks & buildlogs

RTX 3090 LLM benchmarks: 70B at 48 GB, and what happens with 32 users at once

Local LLM benchmarks on two RTX 3090s, 8B to 70B, measured with CRUCIBLE, our open source tool. 19 to 145 tokens/s for one user, 2,460 tokens/s across 32 at once, plus cited answers over your own documents.