A research team led by Associate Professor Baiqing Lv at the Tsung-Dao Lee Institute, School of Physics and Astronomy, and Zhangjiang Institute for Advanced Study, Shanghai Jiao Tong University, has made a significant breakthrough in understanding magnetism in kagome metals. In collaboration with Prof. Hong Ding (TDLI), Prof. Yaobo Huang (Shanghai Advanced Research Institute, CAS), Prof. Tian Shang (East China Normal University), Prof. Ziqiang Wang (Boston College), Prof. Lun-Hua He (Institute of Physics, CAS), and others, the team has discovered a novel orbital-selective dual modulation in the antiferromagnetic kagome metal TbTi₃Bi₄. Their findings were recently published in Physical Review X [Phys. Rev. X 15, 031012 (2025)].

Using a combination of angle-resolved photoemission spectroscopy (ARPES), neutron powder diffraction (NPD), scanning tunneling microscopy (STM), and density functional theory (DFT) calculations, the researchers directly observed a striking electronic reconstruction in the antiferromagnetic state below ~20.4 K. They discovered that different electron orbitals “cooperate” in reconstructing the electronic band structure via anisotropic nesting vectors of ~1/3 a* and 0.28 b*. Remarkably, this modulation induces a topological phase transition, leading to the emergence of symmetry-protected Dirac cones at specific momenta.

The modulation is highly orbital-selective and anisotropic: the gap along the a*-direction is three times larger than that along b*, and STM measurements rule out charge density waves as the cause—pointing instead to a magnetic origin. Neutron diffraction reveals a modulated magnetic structure with wavevector (1/3 a*, 0.28 b*, 0), induced by Fermi surface nesting through the RKKY interaction. Crucially, the associated band folding gives rise to Dirac cones at the Brillouin zone boundary, signaling a magnetism-driven topological transition.

Fig. 1. Band reconstruction along the a-direction at the Brillouin zone boundary . (a) ARPES intensity plot along the direction in the paramagnetic state. (b) ARPES intensity plot along the same direction in the antiferromagnetic state. "DP" denotes the Dirac point formed due to band folding. (c), (d) Second-derivative plots corresponding to the paramagnetic (c) and antiferromagnetic (d) states, respectively. q₁ = 1/3 a* indicates the nesting vector.

Fig. 2. Band reconstruction along the b-direction at the Brillouin zone boundary . (a) ARPES intensity plot along the  direction in the paramagnetic state. (b) ARPES intensity plot along the  direction in the antiferromagnetic state. q₂ = 0.28 b* denotes the nesting vector. (c), (d) Corresponding ARPES intensity plots with the Fermi-Dirac cutoff effect removed, for the paramagnetic (c) and antiferromagnetic (d) states, respectively. A comparison of the dashed-box regions reveals partial gap openings at the Fermi level.

Fig. 3. Neutron diffraction results and schematic of the magnetic structure. (a) Powder neutron diffraction patterns in the paramagnetic state (top) and antiferromagnetic state (bottom). Additional diffraction peaks emerge in the antiferromagnetic state, which can be indexed using the nesting vector (1/3, 0.28, 0) obtained from ARPES measurements. (b) Magnetic structure determined from the refinement of powder neutron diffraction data. The magnetic order exhibits a periodicity of approximately 3a along the a-axis and an incommensurate modulation along the b-axis.

DFT analysis pinpoints the Tb 5dₓ𝑧 orbital as the main driver behind the nesting instability and magnetic modulation. The orbital’s strong weight along the A–L₁ direction of the Fermi surface leads to the anisotropic antiferromagnetic ordering observed.

Figure 4. DFT calculation results. Left: Fermi surface at the  plane obtained from DFT calculations. The vectors q₁ and q₂ correspond to the nesting vectors observed in the experiments. Right: DFT-calculated projection of the Tb 5dxz orbital on the  Fermi surface. The size of the circles represents the weight of the Tb 5dxz orbital contribution.

This work also sheds light on the puzzling one-third magnetization plateau seen in TbTi₃Bi₄ under an external magnetic field. The orbital-selective and anisotropic magnetic modulation model explains how the 1/3 a* order remains energetically favorable due to the strong gap anisotropy. A simple magnetic moment flipping model based on a 3a-periodic order reproduces the fractional magnetization plateau and the associated resistivity changes observed in experiments.

Figure 5. Schematic of band reconstruction and the formation mechanism of the fractional magnetization plateau. Left: Illustration of band reconstruction in the antiferromagnetic state. Due to strong anisotropy, the 1/3 a* magnetic order dominates. Right: Schematic of the proposed mechanism for the 1/3 magnetization plateau, based on magnetic moment flipping in a 3a-period antiferromagnetic order.

By demonstrating how orbital degrees of freedom selectively couple with local magnetic moments, this study introduces a powerful strategy for tuning magnetism via orbital engineering—e.g., inserting quasi-one-dimensional chains to enhance anisotropy and induce exotic orders such as fractional magnetization or directional magnetic modulations. This paves the way for designing novel magnetic quantum materials, including robust magnetic skyrmions and unconventional topological phases.

The paper's co-first authors are Ph.D. students Renjie Zhang (TDLI, SJTU & IOP, CAS), Bocheng Yu (ECNU), Yiwei Cheng (TDLI, SJTU & SARI, CAS), and Assoc. Prof. Hengxin Tan (SJTU). The co-corresponding authors are Assoc. Prof. Baiqing Lü (TDLI, SJTU), Prof. Yaobo Huang (SARI-CAS), and Prof. Tian Shang (ECNU). This research was supported by the Ministry of Science and Technology of China, the National Natural Science Foundation of China (NSFC), and the Shanghai Municipal Government.

Article link：

https://journals.aps.org/prx/abstract/10.1103/c3tg-1lxl