[Storage-research-list] Fwd: [Call for Papers] The First International Workshop on Evaluatology (Evaluatology 2024)

[Joan Digney]

Forwarding as the original message was mistakenly deleted.

Joan Digney
List Admin



-------- Forwarded Message --------
Subject: 	[Call for Papers] The First International Workshop on 
Evaluatology (Evaluatology 2024)
Date: 	Mon, 23 Sep 2024 08:36:05 +0800 (CST)
From: 	gaowanling at ict.ac.cn



[Apologies if you receive multiple copies of this CFP]

CALL FOR PAPERS

==========================================================

The First International Workshop on Evaluatology (Evaluatology 2024)

https://www.benchcouncil.org/eva24/index.html

In conjunction with
the 16th BenchCouncil International Symposium on Benchmarking, Measuring 
and Optimizing (Bench 2024)
https://www.benchcouncil.org/bench2024/

* Dec. 4-6, 2024, Guangzhou, China

Important Dates

1-page abstract due date: October 15, 2024, AoE
Notification date: November 1, 2024, AoE
Workshop: December 4, 2024
Extended article due date: January 1, 2025, AoE

Submission Site: https://eva2024.hotcrp.com/

==========================================================


Introduction
---------------------

Evaluatology 2024 is an international workshop dedicated to advancing 
the science and engineering of evaluation.

We cordially invite researchers from various fields to submit their 
work, with a strong emphasis on interdisciplinary research. Whether your 
research focuses on computers, artificial intelligence (AI), medicine, 
education, finance, business, psychology, or other social sciences, all 
relevant contributions are highly appreciated and encouraged.

Evaluatology 2024 follows a post-publication format, where we request 
abstracts of up to one page only. Following the presentation and 
feedback from the audience, the full article will be published in the 
Bench 2024 proceedings or the Tbench journal. To ensure a clear and 
concise focus on evaluation issues, please refer to "A Short Summary of 
Evaluatology: The Science and Engineering of Evaluation, J. F. Zhan, 
BenchCouncil Transactions on Benchmarks, Standards, and Evaluations."
( https://www.sciencedirect.com/science/article/pii/S2772485924000279 )


Call for Papers
---------------------

** Evaluation theory and methodology
- Formal specification of evaluation requirements
- Development of evaluation models
- Design and implementation of evaluation systems
- Analysis of evaluation risk
- Cost modeling for evaluations
- Accuracy modeling for evaluations
- Evaluation traceability
- Identification and establishment of evaluation conditions
- Equivalent evaluation conditions
- Design of experiments
- Statistical analysis techniques for evaluations
- Methodologies and techniques for eliminating confounding factors in 
evaluations
- Analytical modeling techniques and validation of models
- Simulation and emulation-based modeling techniques and validation of 
models
- Development of methodologies, metrics, abstractions, and algorithms 
specifically tailored for evaluations

** The engineering of evaluation
- Benchmark design and implementation
- Benchmark traceability
- Establishing the least equivalent evaluation conditions
- Index design, implementation
- Scale design, implementation
- Evaluation standard design and implementations
- Evaluation and benchmark practice
- Tools for evaluations
- Real-world evaluation systems
- Testbed

** Data set
- Explicit or implicit problem definition deduced from the data set
- Detailed descriptions of research or industry datasets, including the 
methods used to collect the data and technical analyses supporting the 
quality of the measurements
- Analyses or meta-analyses of existing data
- Systems, technologies, and techniques that advance data sharing and 
reuse to support reproducible research
- Tools that generate large-scale data while preserving their original 
characteristics
- Evaluating the rigor and quality of the experiments used to generate 
the data and the completeness of the data description

** Benchmarking
- Summary and review of state-of-the-art and state-of-the-practice
- Searching and summarizing industry best practice
- Evaluation and optimization of industry practice
- Retrospective of industry practice
- Characterizing and optimizing real-world applications and systems
- Evaluations of state-of-the-art solutions in the real-world setting

** Measurement and testing
- Workload characterization
- Instrumentation, sampling, tracing, and profiling of large-scale, 
real-world applications and systems
- Collection and analysis of measurement and testing data that yield new 
insights
- Measurement and testing-based modeling (e.g., workloads, scaling 
behavior, and assessment of performance bottlenecks)
- Methods and tools to monitor and visualize measurement and testing data
- Systems and algorithms that build on measurement and testing-based 
findings
- Reappraisal of previous empirical measurements and measurement-based 
conclusions
- Reappraisal of previous empirical testing and testing-based conclusions


Paper Submission
---------------------

Papers must be submitted in PDF. The review process follows a strict 
dual-anonymous policy per the established Bench conference norms. The 
submissions will be judged based on the merit of the ideas rather than 
the length. After the conference, the proceedings will be published by 
Springer LNCS or TBench journal (Indexed by EI).
At least one author must pre-register for the symposium, and at least 
one author must attend the symposium to present the paper. Papers for 
which no author is pre-registered will be removed from the proceedings.
- The submission must describe novel, unpublished work that is not 
currently under review of any other conference venue or journal.
- All authors and affiliation information must be anonymized.
- Paper must be submitted in printable PDF format (LNCS format).
- Please number the pages of your submission.
- The submission must be formatted for black-and-white printers.
- Please make sure your figures are readable when printed in black and 
white.
- LNCS LaTeX template: https://www.benchcouncil.org/file/llncs2e.zip


Program Committees
---------------------

Program Chairs
Jianfeng Zhan, ICT, Chinese Academy of Sciences, China
Weiping Li, Civil Aviation Flight University of China

Program Associate Chairs
Maodeng Li, Beijing institute of control engineering, China
Wanling Gao, ICT, Chinese Academy of Sciences, China

Technical Program Committees
Hajdi Cenan, AIrt, Croatia
Wei Wang, East China Normal University, China
Jialiang Tan, Lehigh University, USA
Dong Li, Zhongguancun Lab, China
Tong Wu, National Institute of Metrology, China
Davor Runje, AIrt, Croatia
Jianming Tang, Hangzhou Dianzi University, China
Lei Wang, ICT, Chinese Academy of Sciences, China
Xingzhou Zhang, ICT, Chinese Academy of Sciences, China
Jungang Xu, University of Chinese Academy of Sciences, China
Yue Liu, Shanghai University, China
Zhen Jia, Amazon, USA
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/storage-research-list/attachments/20240923/dbb99839/attachment.html>