Case studyPwC Tunisia · final-year engineering project · February to July 2026

Explain My Model: automatic documentation of credit risk models

My final-year project at PwC Tunisia (Risk Services Maghreb): a multi-agent generative AI platform that reads the SAS, Python or R code of a credit risk model and writes its technical and regulatory documentation (IFRS 9, Basel III). The real challenge was not generating text but proving it is correct: every report is checked automatically, scored section by section, then approved by an expert.

Faithfulness to code, 46 evaluations
0.96
Lines of code covered
100%
Success criteria met
4 / 5
  • LangGraph
  • LangChain
  • RAG (BGE-M3, FAISS)
  • DeepEval
  • Llama 3.3 70B
  • Flask
  • SQLite
  • Playwright
  • ReportLab

The subject

A bank must document each of its credit risk models for its auditors and its regulator. IFRS 9 governs how expected credit losses are calculated; Basel III sets the capital the bank must hold. The documentation must let an outsider understand the model, its assumptions and its limits.

Today, a consultant reads by hand programs of thousands of lines, often old, sparsely commented, written in SAS, Python or R, sometimes on proprietary solutions with no public documentation. It is slow and costly, and the documentation drifts away from the code.

The research question: to what extent can generative AI produce, from the source code alone, technical and regulatory documentation usable under regulatory scrutiny? The guiding principle: the documentation must describe what the code does, not what its authors think it does.

Before coding, I set five measurable success criteria (code coverage, faithfulness, reproducibility, effect of review, validated conversions); they are checked against the results below.

Architecture

  1. Code analysisSAS and Python parsers (syntax tree and LLM): blocks, variables, data flow
  2. Source retrievalRAG over IFRS 9, Basel and SAS documentation, then the web, otherwise flagged inference
  3. Writing agentsTechnical documentation, IFRS 9, Basel III, SAS logs
  4. VerificationDeterministic checks and an LLM judge (DeepEval), a score per section
  5. Expert reviewVersioned corrections, then turned into reusable lessons
Multi-agent architecture of the platform
The eight agents, their inputs and the shared memory (diagram from my final-year report)
Expert review loops back to the writing agents: corrections are stored and reused for the next reports.

Simplified diagram, drawn by me.

What I built

  • Eight specialised agents orchestrated with LangGraph: SAS and Python parsers, SAS and Python technical documentation, SAS execution-log analysis, IFRS 9 and Basel III regulatory agents, and an R-to-Python converter that runs the converted code in a sandbox and fixes its errors until it works.
  • A “knowledge cascade”: every explanation cites its source (standard, documentation, web) or states that it is inferred from the code, with a confidence level. The system prefers silence to invention.
  • A regulatory RAG tested, not chosen by reputation: the MiniLM encoder confused “12-month PD” with “lifetime PD”; I replaced it with BGE-M3, which tells them apart, and rebuilt the FAISS index.
  • Language-model routing under quota limits: with no paid access, each task goes to the best-suited free model (Llama 3.3 70B for reasoning, Llama 4 Scout for long texts, GPT-OSS 120B as a critical judge), at zero temperature.
  • Two families of evaluation: what can be checked (does a value exist in the code? does a cited article exist?) is verified in Python with no LLM; what needs interpretation (faithfulness, completeness, citation accuracy) is scored by an LLM judge independent of the generator. Uncertain sections count double in the score.
  • Human review that teaches the system: each correction creates a version; on approval, generalisable corrections become lessons, retrieved by semantic search for later reports, without retraining any model.
  • A Flask web application for the teams: projects, program upload, chat with the agents, PDF reports, separate roles (technical, regulatory), two-factor authentication, and a third-party model check using the user's own key, never stored.

Results

Measurements taken from the application's database, on real runs: 20 SAS, Python and R programs, 31 PDF reports, 244 evaluation measurements, 7 reports reviewed (15 versions).

Faithfulness to code (46 evaluations)
0.96
Technical accuracy (15)
0.93
Cited statements found in the code
99%
Completeness (56)
0.44
Criterion set at the startMeasured resultVerdict
Code coverage above 90%100% of the 421 meaningful lines of 4 programsMet
High, stable faithfulness to code0.96 over 46 evaluationsMet
Strict reproducibilityYes for deterministic checks; generated text variesPartial
Improvement after expert review0.75 → 0.83 on corrected sections (one cycle)Met
Conversions validated by execution3 out of 3, in 2 to 4 attemptsMet
Mean score per evaluation measure
Mean score per measure: deterministic checks on the left, LLM-judge metrics on the right (counts in brackets)

Reading: the documentation is faithful but incomplete. It invents almost nothing (faithfulness 0.96, 99% of cited statements exist), but it leaves things out (completeness 0.44, specificity 0.49): the intended result of instructions that prefer silence to an unsupported claim.

The platform and the diagrams are in French or English, as used during the project.

Limits

Not an audit-ready document. The regulatory documentation links the code to the standard; it does not judge whether the model or its calibration is right. The tests the regulator requires (predicted against observed defaults) were impossible without client data. An expert remains essential, and the platform keeps each report “pending” until one approves it.

A proprietary solution with no public documentation. Of 20 blocks in one technical report, 9 were documented by inference from the code; the report tells the reader so.

Partial reproducibility and an automatic judge. Generated text varies between runs, and substantive correctness is still scored by an LLM, with its known biases.

Free tiers only. Quotas limited the achievable quality: 17 of 32 third-party model checks failed for lack of quota. The results rest on a single review cycle and three conversions.

Confidentiality

This work was done for a company. The code and data remain confidential: this page presents the approach, the validated results and charts from my internship report, with no screenshots of the tools.