Book a call

03 / Pillar Open project

A-200M is a 201.9M-parameter model, built in the open.

Trained from scratch on free GPUs. Apache-2.0. Open to contributors, with the founder.

Validation loss by training stepLower is better
3.03.23.43.605k10k15kSession 2 start: step 1,860, validation loss 3.556Session 2: step 3,903, validation loss 3.268Session 3: step 5,697, validation loss 3.199Session 4: step 7,656, validation loss 3.07Session 5: step 9,677, validation loss 3.059Session 6: step 11,241, validation loss 3.06Session 7: step 12,753, validation loss 3.023Session 8: step 14,766, validation loss 2.922Session 9, pretraining complete: step 15,000, validation loss 2.9713.5562.971
201.9Mparameters
15,000steps, over 9 sessions
3.9Btokens seen
₹0cash on compute

Each point is the validation loss at the end of a training session, from the public training log. Single readings swing by about 0.1, so read the shape, not one dot.

Status

Pretraining finished 12 October 2026.

15,000 steps over 9 sessions on free Kaggle GPUs. Final validation loss about 2.97, perplexity about 19.5. It is a base model. It writes fluent English and is unreliable about facts.

Fine-tuning has not been done. The target job is narrow: find the sentence on a page that states a fact, or say none does.

The test

0 of 150

answers supported by a quote that is really on the page, zero-shot, at the final checkpoint. This is the floor that fine-tuning has to beat. It is not a verdict on the model.

Capability

What it can and cannot do.

QuestionToday
CanContinue English text fluently. Serve as a small base model for research on fine-tuning, tokenization, evaluation and training on limited hardware. Show one complete pretraining run, end to end.
CannotAnswer questions reliably. Cite sources. Follow instructions. Produce valid structured output on request. Extract facts from a page (0 of 150).
BenchmarksNone run. We make no benchmark or comparison claim we have not measured. It is not comparable to large models.

The model

Parameters
201.9M
Architecture
Llama-style decoder: RMSNorm, rotary positions, SwiGLU, grouped-query attention
Shape
15 layers, width 1024, 16 heads, 4 KV heads, context 2,048
Tokenizer
Own BPE, 32k vocabulary
Data
FineWeb-Edu, about 3.93B tokens seen, under one pass
Compute
Free Kaggle GPUs (2x T4). ₹0.

Provenance

  • One dataset, public, with its licence honoured and credited.
  • We added no text written by a language model to the training data. It is a public web crawl, so we cannot promise every page in it was written by a person. Code was written with AI help.
  • Public sources only. Nothing from any private or employer system.
  • Every result is stated with the run that produced it.

Release Not yet published

Licence and release.

Code and weights will be released under Apache-2.0. Nothing has been published yet: no repository, no download, no model on any hub.

In the release

  • The final checkpoint and the tokenizer.
  • The code that built and trained the model, and the data preparation script.
  • The training ledger: every session, its loss, and its mistakes.
  • A model card, the licence and notice files, tests, and the contribution and conduct rules.

Not in the release

  • The training-state file with optimizer state. It is larger and adds nothing for users.
  • Our extraction labels and the company pages they came from. The 0 of 150 result cannot be reproduced from the release alone.
  • Any output written by a hosted model, and any copy of third-party pages.
  • Anything that belongs to a person, employer or customer.

Provenance

  • One dataset: FineWeb-Edu, released under the Open Data Commons Attribution License and derived from Common Crawl. Credited in the model card.
  • Pretraining only. No fine-tuning of any kind was applied to the final checkpoint.
  • We added no text written by a language model to the training data. The data is a public web crawl, so we cannot promise that no web page in it was machine written. The code was written with AI help.
  • No private, employer, customer or login-walled data was used.
  • The weights file matches the hash in the training ledger.

How this was checked

By an audit we ran ourselves in October 2026, reading the data pipeline and scanning the files and history. It is not an outside audit and it is not legal advice. The release is built into a separate folder with no history and scanned again before anyone sees it.

Licence
Apache-2.0, code and weights, on release
Status
Not yet published
Copyright holder
Named in the release

Contribute

Careful beats large. A measured negative result is welcome.

Rules that do not bend

  1. No model-written text in training data. Open-weight teachers only if Apache-2.0 or MIT, run by you, and recorded.
  2. Human labels must be checked by a named human.
  3. Public sources only. Licence written in the pull request.
  4. Share (url, field, short quote, label). Never copies of pages.
  5. No benchmark claim you did not measure. Show the command and the output.
  6. Free-tier friendly. Paid compute is stated up front.

Open an issue first for anything that touches data, training, the tokenizer or evaluation. Sign off every commit with git commit -s to say you have the right to contribute it. By submitting, you license your contribution under Apache-2.0.

Good first tasks

  • A standalone public scorecard for the extraction task.
  • Tokenizer analysis against two public tokenizers.
  • A human-reviewed label set: 10 public company sites, by hand.
  • A perplexity and zero-shot benchmark harness.
  • Hugging Face and safetensors parity, with a logit test.
  • A provenance checker for CI.
Licence
Apache-2.0, code and weights
Sign-off
git commit -s on every commit
Code and weights
Not yet published
Repository
Opens when released

Want a note when it opens? Write to us.