The current and prospective situation of cognitively impaired people entails great human, social, and economical costs. Smart homes can help to maintain in place cognitively impaired people, to improve their autonomy, and accordingly to alleviate the burden put on informal and professional caregivers. The research performed at DOMUS lab aims at turning the whole home into a cognitive prosthetic, especially by providing cognitive assistance. In this process, behavior tracking is a fundamental piece. After sketching the infrastructure, two cognitive assistants are used to illustrate how activity recognition can help to address four kinds of cognitive deficits (initiation, attention, planning, and memory). An experimentation of one of these asistant involving people with intellectual deficiencies is finally shortly described.