In 2000, the Accreditation Council for Medical Education introduced a new initiative that substantively changed the method by which residency programs are evaluated. In this new competency-based approach to residency education, assessment of performance became a main area of interest, and direct observation was offered as a tool to assess knowledge and skills. Despite being an inherent part of medical education as faculty and learners work together in clinical experiences, direct observation has traditionally been an informal and underused assessment method across all specialties. Residents and students report rarely being observed during their educational process, even though they value the experience. Reasons for this include a lack of faculty time, a lack of faculty skills, a potential stressful effect on the learner, and a perceived lack of validation of the assessment. This article examines the literature regarding the use of direct observation in medical education with a focus on validity evidence. We performed a PubMed search of articles pertaining to direct observation, using key words such as direct observation, performance observation, clinical observation, students, and residents. A subsequent search was conducted in known articles, focusing on variations of the term observation in the titles of articles and introducing the concept of clinical competence. In conclusion, direct observation is a unique and useful tool in the assessment of medical students and residents. Assessing learners in natural settings offers the opportunity to see beyond what they know and into what they actually do, which is fundamentally essential to training qualified physicians. Although the literature identifies several threats to its validity as an assessment, it also demonstrates methods to minimize those threats. Based on the current recommendations and need for performance assessment in education and with attention paid to the development and design, direct observation can and should be included in medical education curricula.