👤EgoSelf: From Memory to Personalized Egocentric Assistant

Yanshuo Wang1,2*, Yuan Xu3*, Xuesong Li4, Jie Hong5, Yizhou Wang3, Chang Wen Chen2†, Wentao Zhu1†
1Eastern Institute of Technology, Ningbo, 2Hong Kong Polytechnic University
3Peking University, 4CSIRO, 5University of Hong Kong
* Equal Contribution, † Corresponding Author
Paper Code arXiv
TL;DR overview of EgoSelf

TL;DR: We present EgoSelf, a graph-based egocentric assistant equipped with interaction memory and dedicated personalized learning to model long-term user behaviors and enable user-specific customization.

📝Abstract

Egocentric assistants often rely on first-person view data to capture user behavior and context for personalized services. Since different users exhibit distinct habits, preferences, and routines, such personalization is essential for truly effective assistance. However, effectively integrating long-term user data for personalization remains a key challenge. To address this, we introduce EgoSelf, a system that includes a graph-based interaction memory constructed from past observations and a dedicated learning task for personalization. The memory captures temporal and semantic relationships among interaction events and entities, from which user-specific profiles are derived. The personalized learning task is formulated as a prediction problem where the model predicts possible future interactions from individual user's historical behavior recorded in the graph. Extensive experiments demonstrate the effectiveness of EgoSelf as a personalized egocentric assistant.

🔬Method

Method overview of EgoSelf
Overview of the EgoSelf Framework. EgoSelf is a personalized egocentric assistant framework that constructs a heterogeneous, graph-structured memory of personal interactions. In this memory, nodes represent historical interaction events, involved objects, and persons, while edges encode the temporal, causal, and semantic relations between them. Leveraging this structured memory, the system extracts user-specific habit profiles that summarize long-term behavioral patterns, enabling personalized response generation.

📊Results

Performance comparison on EgoLifeQA
Performance comparison on the EgoLifeQA benchmark. EgoSelf delivers superior performance on tasks involving temporal event recall and relational interaction modeling, demonstrating strong capabilities in temporal understanding and social relationship modeling.
Retrieval and accuracy comparison on 7-day task
Retrieval and accuracy performance comparison between EgoSelf and EgoRag on the 7-day task. The yellow line represents our EgoSelf performance, and the green line represents the compared baseline. The proposed method consistently maintains stable and superior performance across all evaluated temporal durations.

🎬Demo Video

BibTeX

@misc{wang2026egoself,
      title={EgoSelf: From Memory to Personalized Egocentric Assistant},
      author={Wang, Yanshuo and Xu, Yuan and Li, Xuesong and Hong, Jie and Wang, Yizhou and Chen, Chang Wen and Zhu, Wentao},
      year={2026},
      eprint={2604.19564},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2604.19564},
}