Rashomon-PDP

The project

Rashomon-PDP is a python package that allows for the use of the Rashomon Partial Dependence Profile (PDP) framework to aggregate and plot the explanation performance of multiple near-optimal models. It currently integrates with the models in libraries H2O and AutoGluon. It also integrates with DALEX’s tool package, along with having a base module that lets developers integrate their own models into the package. This package allows for data scientists and other academics such as our client to have an effective way of examining the performance of multiple well-performing models. By aggregating the models into the highest-performing “Rashomon Set”, models that perform just as well as the alleged “highest-performing” model aren’t ignored during PDP analysis.

The customer

Our client is Mustafa Cavus. He is a researcher (Phd.) at the Department of Statistics, Eskisehir Technical University in Turkey. He has written a paper: ”Beyond the Single-Best Model: Rashomon Partial Dependency Profile for Trustworthy Explanations in AutoML”, together with Jan N. van Rijn and Przemyslaw Biecek, which describes the Rashomon Partial Dependency Profile formed the basis of our project’s python package. Bi-weekly virtual meetings were held with the client so our team’s work would remain synchronous with his needs.

The team

Product Owner: Laith Agbaria SCRUM Master: CJ Reitter SCRUM Developers: Varun Brahmadin Marit Paul Bing Steens Gideon Mazzola

The timeline main features/issues worked on throughout the project were decided on early into the project. For each of these issues worked on during a given sprint, sub-tasks were formed and divided evenly between members.

Each member generally had specific issues they were always assigned to. After some conflicts emerged during the early sprints, the original roles that members were given were reassigned, allowing for a more efficient workflow. This allowed us to effectively work on the project, maintaining SCRUM compliance throughout and eventually achieving a client-approved MVP by sprint 6 of 9, allowing us to go above and beyond what the client expected.

The technologies

Language:

• Python

Important frameworks and libraries:

• H20, AutoGluon, & DALEX: main AutoML libraries integrated into the package for library usage
• Streamlit: web framework for running the UI of the package

Tooling:

• Ruff: linting and formatting of Python code (also used in CI)
• pytest: automated testing (also used in CI)

Deployment:

• Our package was deployed to PyPi and can be downloaded via pip
• Our package’s UI runs on a local hosting server on users’ devices, powered via Streamlit