OpenML Uploading Interface

The project

The goal of the project is to provide our client with an easy-to-use web application which allows the users of OpenML to contribute their datasets according to the OpenML format.

OpenML lackeid proper tooling for contributors to upload certain datasets, hence we address this problem by implementing a community dataset upload interface which users can use to upload any dataset, and configure it with Croissant metadata. The uploaded datasets are scanned for malware, and reviewed by OpenML experts before uploading to the official OpenML directory.

Our product utilizes a FastAPI backend with a React frontend. Users authenticate via Github and then are granted access to the dataset uploading mechanism.

We directly impact the OpenML community by allowing everyone to utilize a pipeline to safely upload their own datasets, through a transparent mechanism, and with the collaboration of the OpenML experts. The experts are there to guide the user in the upload process, since defining the Croissant metadata can be challenging.

A single path for contributors and experts to move datasets safely into the OpenML ecosystem.

The customer

Jan N. van Rijn is the client for this project and serves as the primary stakeholder responsible for providing requirements and making final acceptance decisions. He is an Assistant Professor at Leiden University within the Leiden Institute of Advanced Computer Science (LIACS) and the Automated Design of Algorithms (ADA) cluster, and he has substantial domain expertise related to OpenML, including doctoral research connected to the platform.

After each sprint we proceed to have a meeting with our Client (Jan) to discuss the results of the sprint and to further plan the direction of the next sprint. After every meeting we would iterate on the product to improve and implement the suggestions from our clients. Overall our communication was structured, we had a clear agenda and a product demo going into each of the meetings, and we have been able to always end the meetings with a sense of direction.

One has to be brave in order to fetch and pull the latest and hope that the working code has not been obliterated.

The team

Our SCRUM approach is to have a total of at least 6 sprints, each lasting 2 weeks. Our SCRUM Master (Alexander) acts as a leader who monitors the SCRUM board and ensures that we are on schedule. The Product Owner (Johannes) defines the vision of the project, manages the product backlog, and prioritizes work. Everyone on the team are developers who create and develop the product incrementally. The principle of our work methodology is that everyone works on the code equally, however there are certain domains where we have assigned people to primarily work on, such as backend and frontend. All work and processes are visible to everyone involved via the SCRUM board and the github repository. Furthermore, we hold frequent checks on progress and work to make necessary adjustments quickly. After each sprint we proceed to have a meeting with our Client (Jan) to discuss the results of the sprint and to further plan the direction of the next sprint.

The most dominant challenge in our team was communication. With many people, it is harder to keep track of what is going on. To fix this, we put a great emphasis on the scrum board and our whatsapp group to actively propagate the latest information, and keep everyone up to date.

As a team, we are the most proud about our ability to create clean and working code. We have thorough CI/CD pipelines that test and lint the code, ensuring all code is uniform. Furthermore we place heavy restrictions on Github, for instance, commit messages have to be in a certain standardized format for a push to be valid. Our deployment pipeline compiles each valid production build on the main branch to a docker file that is easily deployable by anyone. Overall, we try to enforce an environment where clean and working code is the only way forward.

One sprint. One day. Only Gabe can save the play.

The technologies

Languages:

• TypeScript: frontend
• Python: backend

Frameworks and liibraries:

• FastAPI: REST API
• SQLAlchemy + Alembic: ORM and Database
• Tailwind CSS + Radix UI: styling

Tooling:

• Vite
• Vitest: frontend tests
• pytest: backend tests
• Playwright: end to end tests
• Black + Flake8 / Eslint + prettier: linting and formatting

Infra:

• PostgreSQL
• MinIO: file storage
• ClamAV: malware scanning of uploaded files
• Docker Compose: local and production orchestration