GPT-4 reads band gap values from paper sentences to train better predictors

The work

Technical · from the record

GPT-4 was prompted zero-shot to pull materials, property values and descriptors out of single-sentence excerpts that had previously been collected with ChemDataExtractor, with four follow-up prompts used to keep only experimentally measured band gaps of pure single-crystalline bulk materials. The filtered result was 39391 extractions covering 2733 distinct compositions, and on 100 manually annotated sentences the prompt-based extraction reached 99% precision against 81% for ChemDataExtractor. Graph neural networks were then trained on each candidate dataset and compared on a shared hold-out set of 210 materials, using matbench_expt_gap labels as ground truth; the paper reports a 19% reduction in mean absolute error on that test set, attributed in the Results section to the dataset combining the new extractions with matbench_expt_gap. The authors also prompted the LLM to generate and execute code that trained logistic regression, gradient boosted trees and random forest models on the extracted data.

How AI was used

An LLM (GPT-4) was run over sentences released with an existing ChemDataExtractor-generated band gap dataset. A first zero-shot prompt extracted, per sentence, each material together with the property name, value, unit and any material or property descriptors. Four further prompts were then run on each extracted entry to ask whether the property is genuinely a band gap, whether the material is a pure non-doped bulk single crystal, what the chemical formula is, and whether the value was numerically calculated. Rule-based inclusion criteria then kept entries with correct prompt formatting, no evidence of numerical computation, no evidence against pure bulk single-crystalline form, units and property names consistent with band gap, and values between 0 and 20 eV; compositions with several extractions were reduced to their median. Extraction and classification prompts were scored for precision and recall against human annotation of randomly selected sentences. Message-passing graph neural networks with matscholar_el node features, an embedding size of 64 and two hidden layers of size 164 and 64 were trained for 1000 epochs with Huber loss, ensembled as the mean of 10 random initialisations, and fitted separately on each candidate training dataset for comparison by 5-fold cross validation, on a shared hold-out set of materials common to all datasets, and on the matbench leaderboard. Finally, the LLM was prompted in natural language to load and manipulate the extracted data and to generate and execute code training logistic regression, gradient boosted trees and random forest band gap models.