03 / Pillar Open project
A-200M is a 201.9M-parameter model, built in the open.
Trained from scratch on free GPUs. Apache-2.0. Open to contributors, with the founder.
Each point is the validation loss at the end of a training session, from the public training log. Single readings swing by about 0.1, so read the shape, not one dot.
Status
Pretraining finished 12 October 2026.
15,000 steps over 9 sessions on free Kaggle GPUs. Final validation loss about 2.97, perplexity about 19.5. It is a base model. It writes fluent English and is unreliable about facts.
Fine-tuning has not been done. The target job is narrow: find the sentence on a page that states a fact, or say none does.
The test
answers supported by a quote that is really on the page, zero-shot, at the final checkpoint. This is the floor that fine-tuning has to beat. It is not a verdict on the model.
Capability
What it can and cannot do.
| Question | Today |
|---|---|
| Can | Continue English text fluently. Serve as a small base model for research on fine-tuning, tokenization, evaluation and training on limited hardware. Show one complete pretraining run, end to end. |
| Cannot | Answer questions reliably. Cite sources. Follow instructions. Produce valid structured output on request. Extract facts from a page (0 of 150). |
| Benchmarks | None run. We make no benchmark or comparison claim we have not measured. It is not comparable to large models. |
The model
- Parameters
- 201.9M
- Architecture
- Llama-style decoder: RMSNorm, rotary positions, SwiGLU, grouped-query attention
- Shape
- 15 layers, width 1024, 16 heads, 4 KV heads, context 2,048
- Tokenizer
- Own BPE, 32k vocabulary
- Data
- FineWeb-Edu, about 3.93B tokens seen, under one pass
- Compute
- Free Kaggle GPUs (2x T4). ₹0.
Provenance
- One dataset, public, with its licence honoured and credited.
- We added no text written by a language model to the training data. It is a public web crawl, so we cannot promise every page in it was written by a person. Code was written with AI help.
- Public sources only. Nothing from any private or employer system.
- Every result is stated with the run that produced it.
Release Not yet published
Licence and release.
Code and weights will be released under Apache-2.0. Nothing has been published yet: no repository, no download, no model on any hub.
In the release
- The final checkpoint and the tokenizer.
- The code that built and trained the model, and the data preparation script.
- The training ledger: every session, its loss, and its mistakes.
- A model card, the licence and notice files, tests, and the contribution and conduct rules.
Not in the release
- The training-state file with optimizer state. It is larger and adds nothing for users.
- Our extraction labels and the company pages they came from. The 0 of 150 result cannot be reproduced from the release alone.
- Any output written by a hosted model, and any copy of third-party pages.
- Anything that belongs to a person, employer or customer.
Provenance
- One dataset: FineWeb-Edu, released under the Open Data Commons Attribution License and derived from Common Crawl. Credited in the model card.
- Pretraining only. No fine-tuning of any kind was applied to the final checkpoint.
- We added no text written by a language model to the training data. The data is a public web crawl, so we cannot promise that no web page in it was machine written. The code was written with AI help.
- No private, employer, customer or login-walled data was used.
- The weights file matches the hash in the training ledger.
How this was checked
By an audit we ran ourselves in October 2026, reading the data pipeline and scanning the files and history. It is not an outside audit and it is not legal advice. The release is built into a separate folder with no history and scanned again before anyone sees it.
- Licence
- Apache-2.0, code and weights, on release
- Status
- Not yet published
- Copyright holder
- Named in the release
Contribute
Careful beats large. A measured negative result is welcome.
Rules that do not bend
- No model-written text in training data. Open-weight teachers only if Apache-2.0 or MIT, run by you, and recorded.
- Human labels must be checked by a named human.
- Public sources only. Licence written in the pull request.
- Share (url, field, short quote, label). Never copies of pages.
- No benchmark claim you did not measure. Show the command and the output.
- Free-tier friendly. Paid compute is stated up front.
Open an issue first for anything that touches data, training, the tokenizer or evaluation. Sign off every commit with git commit -s to say you have the right to contribute it. By submitting, you license your contribution under Apache-2.0.
Good first tasks
- A standalone public scorecard for the extraction task.
- Tokenizer analysis against two public tokenizers.
- A human-reviewed label set: 10 public company sites, by hand.
- A perplexity and zero-shot benchmark harness.
- Hugging Face and safetensors parity, with a logit test.
- A provenance checker for CI.
- Licence
- Apache-2.0, code and weights
- Sign-off
- git commit -s on every commit
- Code and weights
- Not yet published
- Repository
- Opens when released
Want a note when it opens? Write to us.