Jekyll2026-09-05T06:53:32+00:00https://pku-pillar-group.github.io/feed.xmlPILLAR GroupThe PILLAR Group is a natural language processing research lab at Peking University. We study large language models, reasoning, LLM agents, and multilingual NLP. Welcome to Lab Notes2025-09-01T01:00:00+00:002025-09-01T01:00:00+00:00https://pku-pillar-group.github.io/blog/2025/welcome-to-lab-notesWelcome to Lab Notes, the blog of the PILLAR Group at Peking University.

We plan to use this space for informal write-ups that do not fit into papers:

  • Paper deep-dives — behind-the-scenes details, ablations, and negative results from our published work
  • Tutorials and reading notes — on topics like large language models, reasoning, and agents
  • Lab updates — new members, talks, and open-source releases

Stay tuned — and if you are interested in working with us, check out the join us page.

]]>A Practical Checklist for Evaluating LLM Reasoning2025-08-20T01:00:00+00:002025-08-20T01:00:00+00:00https://pku-pillar-group.github.io/blog/2025/evaluating-llm-reasoningEvaluating reasoning in large language models is trickier than it looks. Here is a checklist we have found useful in our own projects.

1. Report more than accuracy

Pass@1 alone hides a lot. Pair it with pass@k, self-consistency over multiple samples, and variance across seeds.

2. Watch for data contamination

If the benchmark is public, assume it is in the training data. Prefer held-out or newly constructed test sets, and report perplexity-based contamination checks where possible.

3. Separate reasoning from retrieval

A model that answers correctly may be reciting, not reasoning. Include adversarial variants — same question, perturbed numbers or entities — to test whether the reasoning chain actually transfers.

4. Check the chain, not just the answer

Final-answer agreement between chain-of-thought and direct answering can reveal when the rationale is post-hoc. Report both.

This is placeholder content — replace it with a real post.

]]>