Local AI Chatbot

About Local AI Chatbot

How Does Browser-Based AI Work?

Traditional AI chatbots send your messages to a cloud server where a large model processes them. This tool takes a different approach: it downloads a quantised (compressed) model directly to your browser and runs inference on your device's GPU using the WebGPU API. WebGPU is a modern browser standard developed by the W3C GPU for the Web Working Group that gives web applications direct access to GPU compute capabilities.

The models used here are compressed using INT4 quantisation, which reduces the memory footprint by roughly 4x compared to their full-precision (FP16) versions. This is what makes it possible to fit a 3.8 billion parameter model into a 2 GB download. According to benchmarks from the MLC AI team, WebLLM retains up to 80% of native GPU performance when running inside a browser tab - an M3 Max chip can run Phi-3.5 Mini at around 71 tokens per second through WebGPU, which is close to what you would get running the same model natively.

Worked example: When you select the SmolLM2 1.7B model and click "Load Model", the browser downloads roughly 1 GB of quantised weights from a CDN. These weights are stored in the browser's Cache Storage API, so the next time you visit, the model loads from local cache in a few seconds. Once loaded, you type a message, the model tokenises it, runs it through the transformer layers on your GPU, and streams the output back token by token - all without any network request leaving your machine.

Available Models

Model	Parameters	Download Size	Context Window	Quality	Hardware Needed
SmolLM2 360M	360 million	~200 MB	2,048 tokens	Basic - simple questions, short completions	Most modern devices with WebGPU support
SmolLM2 1.7B	1.7 billion	~1 GB	2,048 tokens	Good - coherent conversations, basic reasoning	Mid-range GPU or Apple Silicon
Phi-3.5 Mini	3.8 billion	~2 GB	128,000 tokens	Best - stronger reasoning, multilingual support	Dedicated GPU with 4+ GB VRAM or recent Apple Silicon

SmolLM2 is a family of compact language models from Hugging Face, trained on approximately 11 trillion tokens of web text, code, and instruction-following data. The 1.7B variant outperforms other models in its size class, including Qwen2.5-1.5B and Llama 3.2 1B, on benchmarks like HellaSwag (68.7) and ARC (60.5). Phi-3.5 Mini is a 3.8 billion parameter model from Microsoft, released under the MIT licence. It was trained on 3.4 trillion tokens and supports 23 languages including Arabic, Chinese, French, German, Japanese, Korean, and Spanish. Despite having only 3.8B parameters, it is competitive with much larger models like Llama 3.1 8B and Mistral 7B on standard benchmarks.

Cloud AI vs Local AI

Aspect	Cloud AI (ChatGPT, Claude)	Local AI (this tool)
Model size	Hundreds of billions of parameters	360M - 3.8B parameters
Response quality	Very high	Basic to moderate
Privacy	Data sent to provider servers	Nothing leaves your device
Cost	Subscription or per-token API fees	Free (uses your own hardware)
Internet required	Yes, for every message	Only for initial model download
Speed	Fast (powerful server GPUs)	Varies by hardware (5-70+ tok/s)
Offline capable	No	Yes, after first download

The privacy angle is increasingly relevant. Cisco's 2025 Data Privacy Benchmark Study found that data leaks from generative AI are the leading security concern for organisations, cited by 34% of respondents (up from 22% in 2024). Running models locally eliminates this risk entirely - your prompts and responses stay on your hardware. For testing prompt structures before sending them to a paid API, the prompt template builder can help organise your inputs.

WebGPU Browser Support (as of April 2026)

As of November 2025, all major browsers ship WebGPU by default, a milestone announced by Google on 25 November 2025. Here is the current support status:

Browser	WebGPU Support	Notes
Chrome 113+	Yes (since April 2023)	Best support, recommended. Android support from Chrome 121+
Edge 113+	Yes	Same Chromium engine as Chrome
Firefox 141+ (Windows)	Yes	Enabled by default on Windows from Firefox 141
Firefox 145+ (macOS ARM)	Yes	Enabled by default on Apple Silicon Macs from Firefox 145
Safari 18+	Yes	macOS, iOS, iPadOS, and visionOS
Mobile browsers	Partial	Android Chrome 121+; iOS Safari 18+

Firefox on Linux is expected to ship WebGPU support later in 2026. If your browser does not support WebGPU, the tool will display a clear message with browser recommendations.

Performance Expectations by Hardware

Hardware	SmolLM2 360M	SmolLM2 1.7B	Phi-3.5 Mini
Integrated GPU (Intel/AMD)	10-20 tok/s	3-8 tok/s	May not load
Apple M1/M2	20-30 tok/s	10-15 tok/s	5-10 tok/s
Apple M3/M4	25-35 tok/s	15-20 tok/s	8-15 tok/s
RTX 3060 / 4060	25-40 tok/s	15-25 tok/s	10-18 tok/s

The first load downloads the model weights, which are then cached by your browser. Subsequent sessions load from cache in seconds rather than minutes. The tool displays a live tokens-per-second counter so you can gauge your hardware's actual throughput. For estimating whether a specific model fits your GPU before downloading, the AI model size calculator shows VRAM requirements for popular models.

Good Use Cases for Local AI

Use Case	Why Local Works Well
Sensitive or private conversations	Nothing leaves your device - ideal for confidential topics
Offline writing assistance	Works without internet after initial download
Learning about LLMs	Experiment with prompts without any cost or rate limits
Quick brainstorming	Instant responses for simple ideas and drafts
Testing prompt structures	Draft and iterate on prompts before sending to a paid API
Air-gapped environments	Works on machines with no internet after the initial cache

Tips for Getting the Best Results

Smaller models respond best to clear, direct prompts. Keep your instructions short and specific. For example, "List 5 breakfast ideas using eggs" will produce better output than a long, multi-part request. If the model gives a weak answer, try rephrasing rather than asking it to "try again" - smaller models do not handle meta-instructions as well as larger ones.

The system prompt field lets you set the model's behaviour. Setting it to something like "You are a coding assistant. Reply with code only, no explanations." focuses the output and reduces filler. SmolLM2 models have a 2,048-token context window, so very long conversations will start to lose earlier context. If responses start seeming disconnected, clear the chat and start a fresh conversation with the key context restated.

Phi-3.5 Mini has a much larger 128K context window, so it handles longer conversations better. It also supports 23 languages, making it the better choice if you need non-English output. For checking the exact token count of a prompt before sending it to a cloud model, the AI token counter runs real tokenisers for GPT-4, Claude, and other popular models.

What Is Quantisation and Why Does It Matter?

Full-precision AI models store each parameter as a 16-bit or 32-bit floating point number. A 3.8 billion parameter model at FP16 would need roughly 7.6 GB of memory - too large to fit comfortably in a browser. Quantisation compresses these values down to 4 bits (INT4), cutting the memory requirement by about 4x. The trade-off is a small reduction in output quality, but for most conversational tasks, the difference is barely noticeable.

The models in this tool use MLC (Machine Learning Compilation) to convert the quantised weights into WebGPU-compatible shader programs. This compilation step happens once per model version and is handled by the MLC AI team before distribution. The result is a set of binary files that your GPU can execute directly through the WebGPU API, without needing Python, CUDA, or any other local runtime. WebLLM has over 17,000 stars on GitHub and supports a wide range of model families including Llama, Phi, Gemma, Mistral, and Qwen.

Limitations to Be Aware Of

These are small models designed for basic tasks. They will not match the quality of cloud models like GPT-4 or Claude for complex reasoning, long-form writing, or nuanced tasks. Common limitations include occasional repetition, difficulty following multi-step instructions, and factual errors on obscure topics. Treat the output as a rough draft rather than a final answer.

The 2,048-token context window on SmolLM2 models means you cannot paste in very long documents for summarisation. Once the conversation exceeds the context window, earlier messages are dropped, which can cause the model to lose track of what was discussed. For longer conversations, Phi-3.5 Mini with its 128K context window is a much better choice. For tasks that genuinely need a large, capable model, cloud services like ChatGPT or Claude remain the better option.

GPU memory is also a constraint. If your device does not have enough VRAM, larger models may fail to load or run extremely slowly. The tool will show an error message if loading fails. Start with the smallest model (SmolLM2 360M) to test your hardware, then move up if it runs well. To check VRAM requirements for any model before downloading, use the AI model size calculator.

Mobile device support is limited. While some Android Chrome builds and iOS Safari 18+ have WebGPU, performance on phones and tablets is generally too slow for a good experience. Desktop browsers on laptops and desktops with a dedicated or recent integrated GPU give the best results. Battery-powered laptops may also throttle GPU performance to save power, so plugging in can noticeably improve token generation speed.

Sources

• MLC AI - WebLLM GitHub Repository
• Hugging Face - SmolLM2 1.7B Instruct Model Card
• Microsoft - Phi-3.5 Mini Instruct Model Card
• web.dev - WebGPU Supported in Major Browsers
• Cisco - 2025 Data Privacy Benchmark Study
• Can I Use - WebGPU Browser Support