COVID-19 pandemic forced governments to enact mobility restrictions to contain the spread of infection. Elderly, especially those with co-morbidities, are highly susceptible to infections. Hence, reverse quarantine measures have been enacted to protect the elderly. The barriers in social connectedness have an impact on the mental and physical well-being of the elderly. Stress, anxiety, anger, confusion, depression, and suicidal tendencies are reported among the elderly kept in social isolation. Apart from the elderly, their family caregivers also go through similar physical and psycho-social stress. Many times, the caregivers are also having additional responsibilities of childcare and homeschooling during the pandemic time. The authors outline the various intricate issues and dimensions of the pandemic's effect on the elderly and their caregivers. Various feasible strategies to address these multidimensional problems faced by the elderly and their caregivers during the pandemic time, like mobile-based mindfulness meditation training, spiritual support, telephone-based psychotherapy, telehealth services, physical exercise promotion, and social support measures, are recommended.