Evaluators reports
This page were completed 2000-06-13 kl 15.00
Proposed terminology
1 - Outstanding/Excellent. Research at a very high international level; of great international interest with broad impact and with publications in internationally leading journals and/or leading conferences; the researchers are among the leading in the field.
2 - Very good. Research at a high international level; of international interest with impact within the sub-field and with publications in internationally leading journals and/or leading conferences; the researchers are among the leading in the sub-field.
3 - Good. Research at a good international level with publications in internationally well-known journals and/or conferences; the researchers have a good international reputation within the sub-field.
4 - Fair. Research that only partly is of good international standard and only partially published in well-known international journals and/or conferences.
5 - Poor. Research of insufficient quality.
General comments
Just as a side comment, these
proposals
are more at simple project level proposals than fully fleshed out research
proposals.
Well, this is not a critique per se, though it might be helpful to start
raising the bar of RT
research and the process of getting projects funded. Its a great thing to
have the current
momentum and having a large number of folks involved, it will make a much
stronger and lasting
impact if we put our quality level at par or higher than conventionally
accepted criteria.
General Comments
I found that these research proposals are, for the most part, not
really effective as research proposals. They typically say,
we will work on problem A. But they don't indicate what the
fundamental scientific or engineering issues exist within that
problem A. The proposals also do not describe any novel ideas
that the authors might have. The state of the art is mostly
ignored or briefly mentioned without explaining how that
relates to their own work. Example, "we want to develop a robot
language." Well, why is it going to be different than all those
other robot languages that exist or were proposed but failed?
The research approach is often too brief and not justified,
and there is a lack of thought
placed in experiments (state hypothesis, and what experiments
will be run to confirm or refute the hypothesis).
So, my comments below are dealing with these proposals relative
to each other rather than on some absolute scale.
1) Proposals 5 and 21 were not within my area of expertice but I reviewed
them. Please give other reviews a higher weight on these proposals.
2) Proposal 19 stood out compared with the other ones. It was the only one
with sufficient technical details. However, most of these details were
communicated within an appended tech report. I am not sure the other
applicants knew that they could attach such a report. This skews the
evaluation. Tech reports should either be permitted or removed.
(Nonetheless, the proposal deserves to be funded.)
Project evaluations
- 0005_04
- Techniques for Module-Level Speculative Parallelization on Shared-Memory Multiprocessors
- Per Stenström, Lennart Pettersson,
- Evaluator 1
Project: ARTES project 0005-4
Title: Techniques for Module-Level Speculative Parallelization on Shared-Memory Multiprocessors
PI: Per Stenstrom
Ranking: Very Good
Comments:
This paper addresses one of the current hot topics in shared-memory
multiprocessors: thread-level speculation. Thread-level speculation has
the potential to greatly simplify the development of parallel programs.
To date, there have been two main approaches to developing parallel programs.
First, one can write a sequential program (or take an existing one), and
then use a parallelizing compiler to automatically parallelize it. This
has proven to work well only in very limited domains (e.g., dense linear
algebra) because the compiler must be able to identify all potential data
and control dependences, and normally can only parallelize the computation
when it can prove that there are no possible dependences. This tends to
be extremely conservative, because often dependences do not exist, even
though the compiler cannot prove otherwise. The second approach is to
have the programmer explicitly parallelize an application. This approach
has proven much more effective for irregular applications, like sparse
matrix codes, and symbolic programs such as databases, but is much
more error prone because programmers are naturally less rigorous than compilers.
Thread-level speculation has the potential to help simplify
parallel programming by allowing programmers and/or compilers to focus
on the *likely* case rather than the *worst* case. Run-time support
is provided to help detect--and recover--when dependences actually exist,
rather than requiring that they be conservatively identified apriori. This
approach has been investigated and found promising for several classes of
numerical applications and with varying levels of hardware support.
The novel aspects of this proposed research are two fold. First, the
investigator intends to evaulate the applicability of these techniques to
Java applications. To the best of my knowledge, this has not been done
before and is an increasingly important area. Second, they focus on
software-centric speculation techniques. Because the overheads are higher in
software, the granularity of parallelism must be larger to compensate.
It would be a significant contribution if they could show that
software-centric techniques perform sufficiently well to achieve good
performance for this applications.
In summary, I think this research is interesting and relevant. The
principle investigator has an excellent track record. I strongly
support this proposal.
- Evaluator 2
PROJECT DATA
ARTES project 0005-4
Name: Techniques for Modul-Level Speculative Parallelization
Applicants: Per Stenstrom
Overall rating: Outstanding
OVERVIEW
The project aims at finding more parallelism in realtime applications,
such as the control software of a telephone exchange, through software-
based speculative execution. This is a continuation of a project that
lost its funding.
APPROACH
In order to explore more parallelism in a program, tasks that can
not be guaranteed to be independent are started in parallel anyhow with
the hope that no dependence will occur during their execution. If a
dependence still occurs, it must be detected and all the side-effect
caused by the speculative part of the execution must be undone. The
work proposed here is to generalize the thread-level parallelization
and applied across consecutive modules. The idea is that little
dependency will be found if the modules have been developed
using sound software practice and few global variables are used.
The primary goal of this project is to determine the upper limit
achievable using ideal simulation conditions.
SCIENTIFIC RELEVANCE OF THE PROPOSAL
This is a fairly new research area. Especially the area of
sotware-based speculative execution. The application of this method
across consecutive modules have never been tried before. If
successful, the methods will be applicable to a large area of existing
software. This work has the potentials of being published in the best
of conferences.
The interest from the sponsor Ericsson lies in their huge investments
in modularized software which has been proven hard to parallelize.
However, it is important that the overhead of the detection and rollback
mechanism can be kept at a minimum. The research group at Chalmers have
reported a 10-fold speedup on a 16-way SMP. If this approach is going
to be successful, an even better speedup must be
achieved, possibly through dependancy-detection supported by hardware.
MERITS AND SKILLS OF THE RESEARCH GROUP
This is one of the best research groups in Sweden. The publication
volume and quality is paralleled by none.
SUMMARY
A great research group that already
Positive results have already been demonstrated during the first phase
of the project.
General and widely applicable research.
Relevant for the supporting partner Ericsson.
- 0005_05
- Distributed Real-Time Systems with Minimal Energy Consumption: Analysis and Synthesis
- Krzysztof Kuchcinski, ,
- Evaluator 1
Kuchcinski Proposal No. 5
This proposal is very relevant to ARTES and has great potential for Industrial
use and collaboration. The problem of real-time scheduling and energy
minimization will become more important in the future. The proposal fails
to give any ideas on how they will do the scheduling. For example, what
heuristics will they try? The problem seems quite limited and somewhat easy
if it is a static scheduling problem. In other words can't I just create some
search heuristic or use simulated annealing to create "schedules?" What are
the research questions? If there was dynamic real-time scheduling with
minimization of energy then this would be much more challenging. Author
should have already chosen some real life examples to show that he thought
about the problem. In summary, the problem is good, but
no details on the possible solution.
Overall: Good
- Evaluator 2
Proposal: Artes 5-5
Title: Distributed Real-Time Systems with Minimal Energy Consumptions ...
Authors: Kuchcinski
Description: The authors ask for funding for the development of a
tool that constructs a feasible schedule taking while trying to
minimize energy consumption through dynamic changes of the processor
frequency. The tool work semi-automatic, i.e., an automated loop
checks the feasibility of a schedule while fine-tuning and voltage
scheduling are user-assisted.
Evaluation:
The proposal addresses an interesting problem. I am not aware of any
tool addressing a similar problem for *distributed* *heterogenous*
systems. The references seem to indicate that other work has yielded
results for uni-processors. However, the authors fail to show why
their approach is linked to distributed systems or heterogeneous
environments. They mention a project on partitioning and
scheduling at Lund University but they do not explicitly say if they
want to use their results for this study.
Some other review (by someone with a firm background on the topic)
should determine if this proposal is funded. To me, the topic sounds
good, the description lacks details but comes close to being
acceptable. Two years are realistic for one student, IMHO. (But this
proposal seems stronger than 5-21!)
Resources the authors should check out
RTAS'98 WIP: Pering/Brodersen (UC Berkeley): Energy Efficient Voltage
Scheduling for Real-Time Operating Systems
Industrial Relevance:
no company indicated, just general "believe" of relevance (which is plausible)
Further Comments:
This is not my area of specialization. My opinion has a medium to low
confidence interval :-)
- 0005_07
- TATOO-Test and Testability of Distributed Real-Time Systems
- Henrik Thane, ,
- Evaluator 1
ARTES Project 0005-7
TATOO: Test And Testability of Distributed Real-time Systems
Henrik Thane
Overall evaluation: Excellent/Very Good
This proposal is for the extend support of the TATOO project for
additional 24-30 months period beginning August 2000. The TATOO
project deals with the important fundamental problem of testing and
monitoring of distributed real-time systems (DRTS). The project team
has developed novel methods for achieving deterministic testing,
debugging and monitoring of DRTS.
The proposed work is to implement prototype tools and evaluate them
through collaboration with an industrial partner, as well as extend
and refine the theory/notion developed for test coverage.
The result of the project will be quite useful and could serve as
foundational work in the area of systematic testing of DRTS. The
tools and methodologies developed will be useful to industries which
build DRTS.
- Evaluator 2
Tatoo Proposal No. 7
This proposal is very different in that it is an extension to a previously
funded one. The previous proposal is included. The problem being worked on
is very important; there is Industrial
support to continue funding; and a number
of results have been produced. The problem is that without going off an
reading those referenced papers I don't know what the results are. The new
proposal wants to continue the work, but there is not a clear statement of what
was done in phase I and what phase II will accomplish. As far as I can tell
the basic approach tires to enumerate all possible orders, but I'm confused
as to how this reproduces the previous execution -- sure somewhere in the
complete ordering the execution that did occur exists, but how do I know
which one? Do you test all of them? This seems impractical. I am probably
missing some important point, but the brevity of the description leaves me
confused as to the proposed solution itself.
Overall: Fair
- 0005_09
- Support for Real-Time 3-D Graphics on Future Mobile Terminals under Energy/Area Constraints
- Per Stenström, Fredrik Dahlgren,
- Evaluator 1
Stenstrom - 3D Graphics Proposal No. 9
This proposal identifies a good problem, a good approach, a novel idea and
demonstrates industry interest. The expected results seem to be too
conservative. Methods and principles while important don't necessarily
produce any new results, i.e., new low power architectures. It was not
clear why there are no plans to actually fabricate solutions rather than just
evaluating via simulations. For example IRAM and CRUSOE projects actually
manufactured products. Also, while this work obviously fits well in the
ARTES program, the proposal only says that explicitly without identifying more
precisely the issues and types of solutions made necessary by real-time
concerns.
Overall: Very Good
- Evaluator 2
ARTES: 0005-9
TITLE: Support for Real-Time 3D Graphics on Future Mobile
Terminals under Energy/Area Constraints
AUTHORS: P. Stenstrom, F. Dahlgren
GOAL
The proposal aims at investigating the design principles of
a general purpose single-chip architecture to provide high
and predictable performance for multimedia applications,
under the constraints of chip area and power dissipation.
COMMENTS
Providing support for 3D graphics on mobile terminals is
important for future distributed architectures and has
certainly great industrial relevance. Moreover, the proposal
is well structured, the project plan is detailed, and the
proponents are qualified to do what they promise.
EVALUATION SUMMARY (POOR, FAIR, GOOD, VERY GOOD, EXCELLENT)
-------------------------------------------------------------
Scientific Relevance: GOOD
Industrial Relevance: EXCELLENT
Technical quality: VERY GOOD
Appropriateness: GOOD
-------------------------------------------------------------
Suggestion: ACCEPT
- 0005_16
- Switched Real-Time Communication for Industrial Applications
- Magnus Jonsson, Bertil Svensson,
- Evaluator 1
Project: ARTES project 0005-16
Title: Switched Real-Time Communication for Industrial Applications
PI: Magnus Jonsson and Bertil Svensson
Ranking: Very Good
Summary:
This project proposes to investigate real-time extensions to switched
Ethernet to provide the service guarantees needed for industrial applications.
Although the Ethernet protocol was originally designed for losses due
to collisions on the shared coax cable, higher speed implementations now
use switches which have the potential to completely eliminate packet loss.
The investigators propose to provide service guarantees for some communication
streams, while making no changes to other higher-level protocols (e.g., TCP/IP).
The latter is particularly important to allow web-based maintenance functions
to co-exist with real-time traffice.
Comments:
This work seems interesting and relevant. Ethernet is and will continue to
be much cheaper than ATM. Finding a way to provide quality of service
guarantees while maintaining compatibility will be a challenge; however
the results will be quite important and practical if the accomplish it.
Most switches and some adaptors provide at least limited ability
to be reprogrammed, which makes this approach quite feasible.
The prior work that that is referenced in the proposal focuses on multimedia
streams, rather than industrial applications. The service guarantees are
generally less strict for multimedia, hence there is room for additional
innovation to support industrial applications.
I like this proposal quite a bit.
- Evaluator 2
ARTES: 0005-16
TITLE: Switched Real-time Communication for Industrial Applications
AUTHORS: M. Jonsson, B. Svensson
GOAL
The objective of this project is to analyze how methods for
supporting real-time traffic can be implemented on cheap and
simple technology, like Ethernet, for developing of industrial
embedded systems.
COMMENTS
The project addresses a very interesting issue, that could
have a strong impact on the development of future industrial
embedded systems. Reducing collisions using cheap LAN switches
which support real-time services is now feasible and worth to
investigate.
Moreover, the proposal is well written, the motivations are
clear and the intermediate goals are described in detail.
Also, the proponents have the right expertize to achieve the
expected results.
EVALUATION SUMMARY (POOR, FAIR, GOOD, VERY GOOD, EXCELLENT)
-------------------------------------------------------------
Scientific Relevance: GOOD
Industrial Relevance: EXCELLENT
Technical quality: VERY GOOD
Appropriateness: VERY GOOD
-------------------------------------------------------------
Suggestion: ACCEPT
- 0005_18
- Pre-Implementation Analysis of Distributed Control Systems - PICADOR
- Martin Törngren, Ola Redell, Jan Wikander
- Evaluator 1
Picador - Torngren Proposal No. 18
This proposal addresses a very difficult and important problem. There is
actually a lot more available than referenced. Since the expected outcome
will be a method and algorithms for pre-implementation temporal analysis,
the author should build upon previous work. For example, can they improve
heuristics by Ramamritham, or the MARS work, or the work on using
simulated annealing by DiNatale? Why not combine and enhance these
heuristics rather than starting over and maybe ending up with the tool
that is not even as good as these approaches.
Overall: Very good. I would recommend funding this project.
- Evaluator 2
ARTES: 0005-18
TITLE: Pre-Implementation Analysis of Distributed Control
Systems - PICADOR
AUTHORS: M. Torngren, O. Redell, J. Wikander
GOAL
The goal of this project is to develop methods for timing
analysis of distributed real-time control systems, in order
to provide tight estimates of timing behaviour such a jitter
and control delays.
COMMENTS
The problem of providing a tight estimate of timing parameters
in a distributed application is a difficult but important
problem (both for academia and industry), because it would
allow to predict the timing behaviour of tasks, so providing
an off-line guarantee of end-to-end constraints.
The proposal is well organized, objectives are clearly stated,
milestones identified, and industrial relavance is well
described in the problem statement. Finally, proponents have
the right expertise to do the job.
EVALUATION SUMMARY (POOR, FAIR, GOOD, VERY GOOD, EXCELLENT)
-------------------------------------------------------------
Scientific Relevance: GOOD
Industrial Relevance: GOOD
Technical quality: VERY GOOD
Appropriateness: VERY GOOD
-------------------------------------------------------------
Suggestion: ACCEPT
- 0005_19
- RATAD - Reliability and Timing Analysis of Distributed Systems
- Hans Hansson, Christer Norström,
- Evaluator 1
ARTES Project 0005-19
RATAD: Reliability and Timing Analysis of Distributed Systems
H. Hansson and C. Norstroem
Overall evaluation: Very good
The authors propose a novel way of analysis for real-time systems that
combines low-level reliability and high-level schedulability analysis
within the same framework. Both kinds of analysis have been employed
in separation by the real-time community for many years. As the
authors aptly point out, schedulability analysis performed without
regard to reliability information leads to sub-optimal results during
design. An approach that would combine the two kinds of analysis in a
workable manner is therefore highly desirable. The topic is therefore
very relevant for the goals of the ARTES project and the current call
for proposals.
The paper provided by the authors in the appendix gives a motivating
case study for the proposed approach. Although the analysis performed
in the case study is highly tailored to the chosen application, it is
a good starting point for the research and may lead for the
development of more general methods.
The topic appears to be a manageable project for a graduate student
under quality supervision. The level of funding requested by the
authors is therefore adequate for the work proposed.
- Evaluator 2
Proposal: Artes 5-19
Title: RTAP
Authors: Hansson ...
Description: The authors ask for funding to integrate stochastic and
deterministic models within real-time systems. Fault models from the
stochastic side should be integrated with deterministic scheduling
models to give a probabilistic approach to timing analysis, which
integrates nicely with common engineering requirements for system
safety.
Evaluation:
This work should definitely be funded. It addresses a central problem,
i.e., the lack of a probabilistic approach to timing analysis. It is
well proposed and a preliminary technical proposal shows convincingly
how the problem can be tackled, both formally and methodically. The
requested funding (1 student, 2 years) seems most appropriate.
Industrial Relevance:
While the topic itself seems relevant to industry, the proposal fails
to show any concrete industrial ties or interests for this work at the
time.
Further Comments:
You claim that distributed scheduling has been solved for real-time
seems very optimistic, given that your cited references constrain the
problem severely, i.e., for an optimal solution you do not get
preemption [Xu] and other approaches typically require a-priori
decisions about task distribution (non-optimal to begin with). A more
careful wording seems to be called for.
The tech report / submitted paper puzzles me. Fig. 1 scenario 1 claims
deadline misses but M2/M3 complete before their respective deadlines
10/20. Where are the missed deadlines? Also, you should definitely address
the problem of different execution time distributions on the
determination of confidence intervals. Next, make clear that the
phasings of interference are your random *variable*! Finally, mention
the constraints on the central limit theorem. Quantify "enough
samples" (which may be difficult but this is important in practice).
- 0005_20
- Extension of Flexible Reliable Timing Constraints
- Gerhard Fohler, ,
- Evaluator 1
Fohler Proposal No. 20
This is an important problem that is being approached in a good manner.
Results have been produced and there is Industrial interest in the work.
A new research project has emerged from the first funded phase. Not
enough details were provided to understand, in depth, the new research
questions to be addressed, except for the motion of transforming reliable
timing constraints into scheduling constraints. Overall, this is a high
quality project, should produce good results and should be funded.
Support letters were missing.
Overall: Very good.
- Evaluator 2
ARTES project 0005-20
Flexible Reliable Timing Constraints (Extension)
G. Fohler
Overall Evaluation: Very Good
An on-going ARTES-funded project concentrates on the treatment of
real-time constraints in the design of embedded real-time systems.
Its approach is based on techniques known as "end-to-end scheduling of
timing constraints," but extends to make more flexible and practical.
As such, it is very relevant for the current call for proposals. The
project has been running for just a few months now, but several
important results have been achieved already. The interest of the
industrial partners in the results of the project promises to increase
the scope of the project even further.
The scope of project has clearly exceeded the current level of funding
by ARTES. One additional graduate student, asked by the proposed
extension, is fully justified by the current state of the project.
The fact that there is a student with experience in the project, who
can start working immediately, makes the extension even more worthy.
- 0005_22
- Real-Time Response and Control of Autonomous Agents
- Nancy E. Reed, ,
- Evaluator 1
Reed Real-Time Response and Control... Proposal 22
This proposal is about real-time AI. The problem with real-time AI
work is that it ignores most of the real-time problems. While
one might be able to use anytime algorithms conceptually, without
linking the high level planning etc to real system level code,
hardware and sensors, very little can truly be said about the
real-time properties of the system. What does efficient mean in
your context? Does it execute in no more than milliseconds?
does it meet a deadline? or is it just polynomial? Is it an
anytime algorithm? Your update to the Pirjanian algorithm seems
like a straightforward use of the anytime idea for this algorithm.
Why is this a significant research/novel idea? What experiments will
you run? How will you run them? Will the results be used on some
real hardware or in simulation?
Overall: Fair.
- Evaluator 2
ARTES: 0005-22
TITLE: Real-Time Response and Control of Autonomous Agents
AUTHORS: N. Reed
GOAL
This project proposes to investigate solutions to the problem
of real-time response and control of autonoumous agents, that
is systems that are embedded in an environment with sensors
for obtaining data and effectors for performing actions.
The project aims at addressing not only the problem of action
selection within timing constraints, but also other related
objectives, such as execution monitoring and long term
planning.
COMMENTS
The proposal addresses an interesting scientific problem, whose
solution has important implications in many application domains,
such as internet searching, economic markets, environment
monitoring, and robotics. Hence, the proposed research has also
a high industrial relevance.
The proposal is well organized, motivations and goals are
clearly stated, milestones are identified.
The only concern of this reviewer is that, although the
proponents have a good expertize in the field of artificial
intelligence and autonomous agents, they never worked on
real-time systems.
EVALUATION SUMMARY (POOR, FAIR, GOOD, VERY GOOD, EXCELLENT)
-------------------------------------------------------------
Scientific Relevance: EXCELLENT
Industrial Relevance: EXCELLENT
Technical quality: FAIR/GOOD
Appropriateness: VERY GOOD
-------------------------------------------------------------
Suggestion: WEAK ACCEPT
- 0005_23
- Embedded Databases for Embedded Real-Time Systems
- Jörgen Hansson, Christer Norström,
- Evaluator 1
Jorgen Hansson - Embedded Databases Proposal No. 23
This proposal addresses a new important, and emerging area. The project
seems very ambitious. While the authors have Industrial partners it is
not clear how they will work with them. Very difficult issues include
the tradeoff-analyses and the composition rules. What are the approaches
in these 2 areas? Will combining components provide good enough performance?
Where and where not has this approach worked? In general these are very
difficult non-functional issues involved with composing components.
Overall: Very good.
- Evaluator 2
ARTES: 0005-23
TITLE: Embedded Dadabases for Embedded Real-TIme Systems
AUTHORS: J. Hansson, C. Norstrom
GOAL
In this project, the proponents want to provide a database
system with minimum memory requirements and necessary
functionality, to be used in industrial embedded applications.
In particular, the project focuses on design tools and
methodologies for developing embedded databases with real-time
services.
COMMENTS
The problem is well formulated and the proposal is technically
sound. The expected results have high industrial relevance
due to the large application domains where embedded databases
can be used.
The project plan is clear and the two academic partners have
an excellent complementary expertize in real-time databases
and real-time systems to investigate the proposed issues.
EVALUATION SUMMARY (POOR, FAIR, GOOD, VERY GOOD, EXCELLENT)
-------------------------------------------------------------
Scientific Relevance: VERY GOOD
Industrial Relevance: GOOD
Technical quality: EXCELLENT
Appropriateness: VERY GOOD
-------------------------------------------------------------
Suggestion: ACCEPT
|