Benchmarking forum – workshop at IPRES 2015

The quality of digital preservation tools is of great importance to the preservation community. However, quality assessment is often done in an isolated way with a lack of systematic and community driven initiatives. Benchmarking is a method of comparing entities to a well-defined standard (benchmark) that has shown itself as a valuable empirical method for evaluating software tools. The successfulness of benchmarking is dependent on the readiness of the community to accept and drive the whole process. This workshop is focused on discussing software benchmarking practices in digital preservation and how these can contribute to improving digital preservation tools.

  1. Benchmarking
  2. Stakeholders and benefits for each group
  3. Workshop agenda
  4. Contributing to the workshop
  5. Outcomes
  6. Workshop organizers


Digital Preservation is characterized by a variety of software tools used to collect and process digital information. High quality tools are required, but quality assessment is often done in an isolated way with a lack of systematic and community driven initiatives.

Benchmarking is a systematic method of facilitating comparisons of software artifacts according to a well defined standard (benchmark).  Benchmarks for Digital Preservation tools paper in the main conference track provides a comprehensive overview, definition and discussion of benchmarks for digital preservation. In short, a software benchmark is a systematic, repeatable method of comparing software tools reliably for a particular purpose. It requires 5 main components: A motivating comparison that specifies what shall be compared with a sufficiently narrow purpose to make meaningful comparisons possible; the specific function to be performed, and a dataset on which it should be performed; ground truth, i.e. knowingly correct answers; and the performance measures to be collected. Note that performance here is not just about speed, but any success measures to be compared.

Benchmarking has shown itself as a valuable empirical method for evaluating software tools. Various fields such as software engineering and information retrieval have reported major benefits from properly established benchmarking initiatives.
For example, TREC , CLEF   and MIREX  have been very effective in catalyzing specific aspects of information retrieval and provided a boost to their research communities.

In the digital preservation field, the full potential of benchmarking has not yet been exploited. However,  several indicators show that the digital preservation community can benefit from establishing benchmarking as a community driven method. To be effective, this needs to be a community-driven process.

Stakeholders and benefits for each group

Practitioners use digital preservation systems and solutions in order to preserve digital material. Being involved in benchmarking initiatives allows practitioners to influence the definition of benchmarks and ensure that the resulting benchmarks reflect their decision needs, use practical terminology, and are relevant. The forum provides initial communication between practitioners and tool developers and allows practitioners to drive the development of tools.

Tool developers  create new and improve existing software tools. By being involved in the benchmarking initiatives they can get effective evaluation methods based on objective metrics and available test data; make testing more effective and its results more visible; promote their solutions; and understand which quality aspects and features are most important and where to invest limited resources for optimal benefit.

Researchers are interested in a variety of aspects. Some develop new tools for digital preservation and can benefit in ways similar to tool developers; some evaluate methods for those tools. Both will benefit from a deeper understanding of the benchmarking process and specific requirements for benchmark components.

For the community, finding a common ground and setting priorities for the systematic evaluation and comparison of software tools will provide an important milestone.

Workshop agenda

The goal of this workshop is to bring together key stakeholders in digital preservation to discuss the needs and challenges of benchmarking in the field, and to define and prioritize initial benchmarks and a future roadmap. We would like to see participants from various domains with a common interest in software quality challenges in digital preservation.

The workshop is planned as a full day event at IPRES 2015. It will take place on Friday , November 6th 2015.

The workshop will be divided into two parts, with a focus on interactive discussion. The first part will consist of

  • An introduction session, where we provide an overview of the workshop together with a brief introduction into benchmarking as a method with illustrations and experiences from other fields.
  • A session where participants present their benchmark proposals and perspectives. Each presentation will be accompanied by time for discussion.

It is expected that these two sessions will fill the time until lunch. After the lunch we will focus on providing a discussion forum.

  • Brainstorm and discuss possible benchmarks. This will build on top of the morning session and will include defining each benchmark according to a common structure with defined components. In the case of a bigger number of submissions, proposals will be grouped and worked on in breakout sessions and each group will present their outcomes.
  • In the final wrap-up session, we plan to discuss the workshop outcomes, reflect on the possibility and challenges of establishing benchmarking in the digital preservation field, and distill specific steps forward and a future roadmap.

Contributing to the workshop

We encourage submission of short position papers (two pages maximum) defining a specific benchmarking need, articulating the challenges behind that benchmark and the benefits expected from implementing that benchmark in the community.

Please submit all your contributions to by  October 15th 2015. Feel free to structure your contributions around the above 5 mentioned components (motivating comparison, function, dataset, ground truth, performance measures).


All  position statements submitted by the participants will be published online before the workshop. Following the workshop, we will write a workshop report for the D-Lib Magazine. Finally, each defined benchmark will be released in a full specification published online to enable wider community involvement.

Workshop organizers

Kresimir Duretec – Vienna University of Technology (Austria)
Artur Kulmukhametov – Vienna University of Technology (Austria)
Christoph Becker – University of Toronto (Canada), Vienna University of Technology (Austria)
Andreas Rauber – Vienna University of Technology (Austria)

We hope to see you at IPRES 2015 in Chapel Hill, North Carolina, USA !

This entry was posted in Uncategorized. Bookmark the permalink.