The first MeetRV³ meetup at IEEE Vis was co-organized by Paul Rosenthal, Wolfgang Aigner, Carlos Scheidegger, and David Koop. The meetups primary goal was to discuss a possible system for rewarding reproducibility in visualization.
As visualization has matured, procedures for verifying and evaluating techniques have improved. While many individuals are making their code and procedures available, there are no established guidelines for reproducibility in our community. Introducing such practices would increase the acceptance of visualization methods in other communities, make contributions more well-grounded, and expedite future innovation by making comparisons and improvements easier.
This important role and the benefits of a strong culture of reproducibility have been discussed over the last years in several panels and workshops in detail. Although there are only a few cases where very special requirements prevent the achieving of basic reproducibility at all, it can be observed that, in current publications, providing reproducibility is often limited to the fraction which is needed to be accepted to a venue. This is also due to the fact that rewards for good reproducibility are very limited.
The goal of the meetup is to recapitulate the discussions and findings of the past workshops and panels regarding this topic. Efforts of other communities to promote reproducibility should be reviewed and discussed by the participants of the meetup to come up with a common direction for the visualization community. We hope to find a consentient way to honor well-reproducible work and strengthen the community in its effort to establish a common culture of reproducibility.
The meetup was very productive and led into a very lively discussion about the different ways to reward good reproducibility. We have collected the findings and feedback in a document and hope to follow the way even further.