<!DOCTYPE html>
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</head>
<body>
<p><font face="Helvetica, Arial, sans-serif">Forwarding as the
original message was mistakenly deleted.</font></p>
<p><font face="Helvetica, Arial, sans-serif">Joan Digney<br>
List Admin<br>
</font></p>
<div class="moz-forward-container"><br>
<br>
-------- Forwarded Message --------
<table cellpadding="0" cellspacing="0" border="0"
class="moz-email-headers-table">
<tbody>
<tr>
<th valign="BASELINE" align="RIGHT" nowrap="nowrap">Subject:
</th>
<td>[Call for Papers] The First International Workshop on
Evaluatology (Evaluatology 2024)</td>
</tr>
<tr>
<th valign="BASELINE" align="RIGHT" nowrap="nowrap">Date: </th>
<td>Mon, 23 Sep 2024 08:36:05 +0800 (CST)</td>
</tr>
<tr>
<th valign="BASELINE" align="RIGHT" nowrap="nowrap">From: </th>
<td><a class="moz-txt-link-abbreviated" href="mailto:gaowanling@ict.ac.cn">gaowanling@ict.ac.cn</a></td>
</tr>
</tbody>
</table>
<br>
<br>
[Apologies if you receive multiple copies of this CFP]<br>
<br>
CALL FOR PAPERS<br>
<br>
==========================================================<br>
<br>
The First International Workshop on Evaluatology (Evaluatology
2024)<br>
<br>
<a class="moz-txt-link-freetext" href="https://www.benchcouncil.org/eva24/index.html">https://www.benchcouncil.org/eva24/index.html</a><br>
<br>
In conjunction with<br>
the 16th BenchCouncil International Symposium on Benchmarking,
Measuring and Optimizing (Bench 2024)<br>
<a class="moz-txt-link-freetext" href="https://www.benchcouncil.org/bench2024/">https://www.benchcouncil.org/bench2024/</a><br>
<br>
* Dec. 4-6, 2024, Guangzhou, China<br>
<br>
Important Dates<br>
<br>
1-page abstract due date: October 15, 2024, AoE<br>
Notification date: November 1, 2024, AoE<br>
Workshop: December 4, 2024<br>
Extended article due date: January 1, 2025, AoE<br>
<br>
Submission Site: <a class="moz-txt-link-freetext" href="https://eva2024.hotcrp.com/">https://eva2024.hotcrp.com/</a><br>
<br>
==========================================================<br>
<br>
<br>
Introduction<br>
---------------------<br>
<br>
Evaluatology 2024 is an international workshop dedicated to
advancing the science and engineering of evaluation.<br>
<br>
We cordially invite researchers from various fields to submit
their work, with a strong emphasis on interdisciplinary research.
Whether your research focuses on computers, artificial
intelligence (AI), medicine, education, finance, business,
psychology, or other social sciences, all relevant contributions
are highly appreciated and encouraged.<br>
<br>
Evaluatology 2024 follows a post-publication format, where we
request abstracts of up to one page only. Following the
presentation and feedback from the audience, the full article will
be published in the Bench 2024 proceedings or the Tbench journal.
To ensure a clear and concise focus on evaluation issues, please
refer to "A Short Summary of Evaluatology: The Science and
Engineering of Evaluation, J. F. Zhan, BenchCouncil Transactions
on Benchmarks, Standards, and Evaluations."<br>
(
<a class="moz-txt-link-freetext" href="https://www.sciencedirect.com/science/article/pii/S2772485924000279">https://www.sciencedirect.com/science/article/pii/S2772485924000279</a>
)<br>
<br>
<br>
Call for Papers<br>
---------------------<br>
<br>
** Evaluation theory and methodology<br>
- Formal specification of evaluation requirements<br>
- Development of evaluation models<br>
- Design and implementation of evaluation systems<br>
- Analysis of evaluation risk<br>
- Cost modeling for evaluations<br>
- Accuracy modeling for evaluations<br>
- Evaluation traceability<br>
- Identification and establishment of evaluation conditions<br>
- Equivalent evaluation conditions<br>
- Design of experiments<br>
- Statistical analysis techniques for evaluations<br>
- Methodologies and techniques for eliminating confounding factors
in evaluations<br>
- Analytical modeling techniques and validation of models<br>
- Simulation and emulation-based modeling techniques and
validation of models<br>
- Development of methodologies, metrics, abstractions, and
algorithms specifically tailored for evaluations<br>
<br>
** The engineering of evaluation<br>
- Benchmark design and implementation<br>
- Benchmark traceability<br>
- Establishing the least equivalent evaluation conditions<br>
- Index design, implementation<br>
- Scale design, implementation<br>
- Evaluation standard design and implementations<br>
- Evaluation and benchmark practice<br>
- Tools for evaluations<br>
- Real-world evaluation systems<br>
- Testbed<br>
<br>
** Data set<br>
- Explicit or implicit problem definition deduced from the data
set<br>
- Detailed descriptions of research or industry datasets,
including the methods used to collect the data and technical
analyses supporting the quality of the measurements<br>
- Analyses or meta-analyses of existing data<br>
- Systems, technologies, and techniques that advance data sharing
and reuse to support reproducible research<br>
- Tools that generate large-scale data while preserving their
original characteristics<br>
- Evaluating the rigor and quality of the experiments used to
generate the data and the completeness of the data description<br>
<br>
** Benchmarking<br>
- Summary and review of state-of-the-art and state-of-the-practice<br>
- Searching and summarizing industry best practice<br>
- Evaluation and optimization of industry practice<br>
- Retrospective of industry practice<br>
- Characterizing and optimizing real-world applications and
systems<br>
- Evaluations of state-of-the-art solutions in the real-world
setting<br>
<br>
** Measurement and testing<br>
- Workload characterization<br>
- Instrumentation, sampling, tracing, and profiling of
large-scale, real-world applications and systems<br>
- Collection and analysis of measurement and testing data that
yield new insights<br>
- Measurement and testing-based modeling (e.g., workloads, scaling
behavior, and assessment of performance bottlenecks)<br>
- Methods and tools to monitor and visualize measurement and
testing data<br>
- Systems and algorithms that build on measurement and
testing-based findings<br>
- Reappraisal of previous empirical measurements and
measurement-based conclusions<br>
- Reappraisal of previous empirical testing and testing-based
conclusions<br>
<br>
<br>
Paper Submission<br>
---------------------<br>
<br>
Papers must be submitted in PDF. The review process follows a
strict dual-anonymous policy per the established Bench conference
norms. The submissions will be judged based on the merit of the
ideas rather than the length. After the conference, the
proceedings will be published by Springer LNCS or TBench journal
(Indexed by EI).<br>
At least one author must pre-register for the symposium, and at
least one author must attend the symposium to present the paper.
Papers for which no author is pre-registered will be removed from
the proceedings.<br>
- The submission must describe novel, unpublished work that is not
currently under review of any other conference venue or journal.<br>
- All authors and affiliation information must be anonymized.<br>
- Paper must be submitted in printable PDF format (LNCS format).<br>
- Please number the pages of your submission.<br>
- The submission must be formatted for black-and-white printers.<br>
- Please make sure your figures are readable when printed in black
and white.<br>
- LNCS LaTeX template:
<a class="moz-txt-link-freetext" href="https://www.benchcouncil.org/file/llncs2e.zip">https://www.benchcouncil.org/file/llncs2e.zip</a><br>
<br>
<br>
Program Committees<br>
---------------------<br>
<br>
Program Chairs<br>
Jianfeng Zhan, ICT, Chinese Academy of Sciences, China<br>
Weiping Li, Civil Aviation Flight University of China<br>
<br>
Program Associate Chairs<br>
Maodeng Li, Beijing institute of control engineering, China<br>
Wanling Gao, ICT, Chinese Academy of Sciences, China<br>
<br>
Technical Program Committees<br>
Hajdi Cenan, AIrt, Croatia<br>
Wei Wang, East China Normal University, China<br>
Jialiang Tan, Lehigh University, USA<br>
Dong Li, Zhongguancun Lab, China<br>
Tong Wu, National Institute of Metrology, China<br>
Davor Runje, AIrt, Croatia<br>
Jianming Tang, Hangzhou Dianzi University, China<br>
Lei Wang, ICT, Chinese Academy of Sciences, China<br>
Xingzhou Zhang, ICT, Chinese Academy of Sciences, China<br>
Jungang Xu, University of Chinese Academy of Sciences, China<br>
Yue Liu, Shanghai University, China<br>
Zhen Jia, Amazon, USA<br>
</div>
</body>
</html>