Leiden Mobile Mocap Lab
The project
The goal of the Leiden Mobile Mocap Lab project is facilitating research or other projects using motion capture technology. It accompanies a toolkit with three cameras and a powerful laptop. The main focus of the project was a sign/movement recording research project. This is an application where a user can record movement of the upper body, with a focus on the hands and face. It was specifically designed to aid research in the field of sign language and movement research, to collect data on individual signs/movements. The recording consists of a video recording and a tracking of the coordinates of set points in the upper body. To accompany this app, the project also contains a website where anyone can upload their research projects using the kit or look into already existing ones. Lastly, it also contains a creative project, which can create paintings by tracking a user’s hands during recording. This part of the project showcases the wide range of capabilities of the motion capture kit. This product helps sign/movement researchers conduct research by providing a toolkit for collecting data on individual signs/movements. The website serves as a platform for researchers, and users, to share and discuss their ideas.
The customer
Our client was the Leiden HANDS!Lab for Sign Languages, a team that hosts research activities in the area of sign language linguistics and Deaf studies. Our main point of contact was Dr. Manolis Fragkiadakis, a researcher and postdoctoral scholar in machine learning, data science, and sign language research at the Leiden University Centre for Linguistics. Dr. Victoria Nyst, a lecturer and associate professor of linguistics at Leiden University, and Dr. Peter van der Putten, an assistant professor and researcher in AI and machine learning at LIACS, served an advisory role.
Communication with the client went smoothly. Our team had bi-weekly meetups with Dr. Fragkiadakis and we kept the rest of the team updated through discord messages and emails. We also met with the rest of the HANDS!Lab, mainly consisting of deaf researchers, to take their perspective on our project into account.
The team
Our team consisted of Matei Canavea, Sofie de Graaf, Sophie Kuo, Sofia Nguyen, Nina Rojewska, and Teodora Tudora. Sophie served as our scrum master, and Matei as our product owner. Further roles pertained to project specifics. Matei was our main website developer. Nina and Teodora developed the creative project. We also divided up different components of the research project. Teodora developed two-handed tracking. Matei developed dynamic movement tracking. Sophie developed refined depth tracking. Sofie and Sofia developed recording of movement. Nina implemented tracking the entire upper body.
We used the SCRUM workflow, with two week sprints. With bi-weekly SCRUM and TA meetings we made sure the tasks assigned to each person were clear and working towards our goals.
Our main challenges had to do with the availability of only three cameras for our team of six. This meant careful consideration of task priorities, sometimes delaying a task until a camera was available for it. This gave us some difficulties at first, but we managed to overcome this by a deliberate divide of tasks among different sprints, balancing camera and non-camera tasks.
We are most proud of finishing the product and implementing all of our client’s requirements to the best of our abilities, even when these evolved throughout the project.
The technologies
Technologies:
- 3 luxonis OAK-D-Pro cameras
- Laptop with strong GPU and 64GB RAM for working with multiple cameras
- DepthAI Python library for OAK cameras
- MediaPipe for Pose estimation