Sage-SciPerception

Scientific perception for language models · 书生·元征

Connect crystals, spectra, biological sequences, and biosignals to language models through trained projectors.

Continuously expanding scientific modalities, tasks, and language model support.

Quick start · 🤗 Weights · Examples · Results · Feedback · 简体中文

Sage Sci Perception maps scientific encoder states, spectral patches, and scientific readouts into a language model's input embeddings, enabling natural-language analysis and structured scientific answers. Explore molecular structure identification, material property prediction, biological sequence analysis, and biosignal classification with task-specific projectors.

Start with the tools in this repository; get the matching weights and configurations from Hugging Face. The download tool selects a checkpoint through the repository index and retrieves its configuration and weights from the same revision. You can check projector loading on CPU, then connect prepared scientific features to a language-model service through the vLLM example. Start with the CPU quick start below.

Examples

Recorded views of scientific inputs, annotations, and task interfaces. Checkpoint evaluations are listed in Results.

Three spectra → molecular structure IR, Raman and UV–Vis with streamed analysis and a recorded prediction.	Protein interface sites Sequence positions and reference residue-level annotations.
Crystal structure and formation energy Atomic geometry and a reference material property.	RNA interface refinement Design revisions guided by experimental feedback.

Quick start

Download a task-specific projector and verify it on CPU. This first step needs no language-model download or GPU.

git clone https://github.com/InternLM/Sage-SciPerception.git
cd Sage-SciPerception
python -m pip install -r requirements.txt
python load_projector.py glm-5.3-flash/cif/jarvis --local-dir ./checkpoints

The selected checkpoint, task configuration, and repository index are saved under checkpoints/. For another task or language-model version, use its ID from the HF component index. The download tool also exposes download_projector(...) for Python workflows.

Run this from the repository directory to check loading on CPU:

from load_projector import load_projector

projector, config = load_projector(
    "checkpoints/projectors/glm-5.3-flash/cif/jarvis", device="cpu"
)
print(config["task"])
print(sum(p.numel() for p in projector.parameters()))
print(config["output_dim"])

Expected output:

cif/jarvis
4314304
4096

Inference with vLLM

The serving path inserts projected scientific vectors into a language model's full templated prompt embeddings and sends them to vLLM through prompt_embeds at /v1/completions. Scientific features and projection are prepared in the client; the language-model service generates the answer.

Start the language-model service

Install vLLM in a compatible inference environment. Use the exact language-model snapshot declared in the selected checkpoint's config.json. Run vLLM in an inference environment with suitable GPUs; set tensor parallelism for that model and hardware. This command shows an eight-way deployment example:

vllm serve /path/to/matching-base-model \
  --served-model-name sage-sciperception \
  --enable-prompt-embeds \
  --tensor-parallel-size 8 \
  --generation-config vllm \
  --max-model-len 8192 \
  --host 127.0.0.1 --port 8000

Generate from prepared crystal features

The crystal client renders the checkpoint's native chat template, looks up text embeddings, inserts scientific vectors, and sends the completed prompt embeddings to the server. Prepare cif_features.pt as a tensor of shape [num_atoms, 64], containing the real CHGNet 0.3.0 atom states specified in cif/jarvis/config.json, in atom order. Once the server is ready, run:

python -m pip install --upgrade transformers
python examples/vllm_cif.py \
  --component ./checkpoints/projectors/glm-5.3-flash/cif/jarvis \
  --base-model /path/to/matching-base-model \
  --features /path/to/cif_features.pt \
  --api-base http://127.0.0.1:8000/v1 \
  --model sage-sciperception

The local language-model directory supplies the tokenizer, configuration, and sharded input embedding table used by the server. The client reads the needed text embedding rows without loading the full language model. For the vLLM prompt embedding interface and release details, see the HF inference guide.

Scientific feature extraction and task-adapted scientific models are separate dependencies; some adapted scientific weights are not yet included in the release. Match each task's scientific_inputs before preparing predictions. The crystal example starts from prepared features, while other tasks require their corresponding input preparation and prompt assembly.

Results

Scores from generated answers using selected task-specific checkpoints. Each checkpoint corresponds to its own task and input representation.

Benchmark / task	Metric ↑	Score	Checkpoint
JARVIS-DFT Formation energy	MAD/MAE	15.03	Files
MPBind PPI interface sites	Per-protein F1 (%)	61.12	Files
QM9S Molecular structure identification	Acc@1 (%)	49.75	Files
SciTS PHU01 ECG classification	Macro-F1 (%)	71.53	Files
GUE splice Splice-site classification	MCC (%)	87.87	Files
SciTS NEU06 Sleep staging	Macro-F1 (%)	72.01	Files
BEACON ncRNA RNA family classification	Accuracy (%)	97.35	Files
SimXRD OL Crystal-system classification	Accuracy (%)	63.60	Files
ATOM3D PSR Protein structure ranking	Spearman (%)	35.36	Files

Metrics differ across rows. MAD/MAE is a ratio where higher is better. MPBind uses the assembly-1 reconstructed evaluation set and mean per-protein F1; PSR uses mean Spearman. These scores do not describe a jointly trained model. The HF model card and each checkpoint's configuration identify the released components.

Framework

A task-specific projector maps scientific representations into the language model's embedding space. During language-mapping training, the scientific models and language model remain frozen while the projector is trained. Some tasks use scientific models adapted beforehand, or combine encoder states with scientific readouts.

Feedback and contributions

For integration help or a new task request, open an issue with the task, input format, and the command or step you want to run. For a bug, include the error output and environment details. Please use a small example that can be shared publicly.

If you would like to contribute an encoder, modality, task, or language-model integration, describe the proposed scientific representation and target task in an issue so the checkpoint format and example can be aligned. Star or watch this repository to follow new checkpoints, examples, and integrations; download releases from Hugging Face.

License details have not yet been specified for this release.

Citation

If you use Sage-SciPerception in your work, please cite:

@misc{sage_sciperception,
  author       = {{InternLM}},
  title        = {{Sage-SciPerception}},
  version      = {0.1.0},
  note         = {GitHub repository},
  year         = {2026},
  url          = {https://github.com/InternLM/Sage-SciPerception}
}