ARTES
-------------------------------------------------

 

Evaluators reports

This page were completed 2000-06-13 kl 15.00

Proposed terminology

1 - Outstanding/Excellent. Research at a very high international level; of great international interest with broad impact and with publications in internationally leading journals and/or leading conferences; the researchers are among the leading in the field.
2 - Very good. Research at a high international level; of international interest with impact within the sub-field and with publications in internationally leading journals and/or leading conferences; the researchers are among the leading in the sub-field.
3 - Good. Research at a good international level with publications in internationally well-known journals and/or conferences; the researchers have a good international reputation within the sub-field.
4 - Fair. Research that only partly is of good international standard and only partially published in well-known international journals and/or conferences.
5 - Poor. Research of insufficient quality.

General comments

Just as a side comment, these proposals are more at simple project level proposals than fully fleshed out research proposals. Well, this is not a critique per se, though it might be helpful to start raising the bar of RT research and the process of getting projects funded. Its a great thing to have the current momentum and having a large number of folks involved, it will make a much stronger and lasting impact if we put our quality level at par or higher than conventionally accepted criteria.


General Comments
I found that these research proposals are, for the most part, not really effective as research proposals. They typically say, we will work on problem A. But they don't indicate what the fundamental scientific or engineering issues exist within that problem A. The proposals also do not describe any novel ideas that the authors might have. The state of the art is mostly ignored or briefly mentioned without explaining how that relates to their own work. Example, "we want to develop a robot language." Well, why is it going to be different than all those other robot languages that exist or were proposed but failed? The research approach is often too brief and not justified, and there is a lack of thought placed in experiments (state hypothesis, and what experiments will be run to confirm or refute the hypothesis). So, my comments below are dealing with these proposals relative to each other rather than on some absolute scale.


1) Proposals 5 and 21 were not within my area of expertice but I reviewed
   them. Please give other reviews a higher weight on these proposals.
2) Proposal 19 stood out compared with the other ones. It was the only one
   with sufficient technical details. However, most of these details were
   communicated within an appended tech report. I am not sure the other
   applicants knew that they could attach such a report. This skews the
   evaluation. Tech reports should either be permitted or removed.
   (Nonetheless, the proposal deserves to be funded.)

Project evaluations

0005_04
Techniques for Module-Level Speculative Parallelization on Shared-Memory Multiprocessors
Per Stenström, Lennart Pettersson,
Evaluator 1
Project: ARTES project 0005-4
Title: Techniques for Module-Level Speculative Parallelization on Shared-Memory Multiprocessors
PI: Per Stenstrom

Ranking: Very Good

Comments:

This paper addresses one of the current hot topics in shared-memory 
multiprocessors: thread-level speculation.  Thread-level speculation has
the potential to greatly simplify the development of parallel programs.
To date, there have been two main approaches to developing parallel programs. 
First, one can write a sequential program (or take an existing one), and 
then use a parallelizing compiler to automatically parallelize it. This
has proven to work well only in very limited domains (e.g., dense linear
algebra) because the compiler must be able to identify all potential data
and control dependences, and normally can only parallelize the computation 
when it can prove that there are no possible dependences.  This tends to
be extremely conservative, because often dependences do not exist, even
though the compiler cannot prove otherwise.  The second approach is to
have the programmer explicitly parallelize an application.  This approach
has proven much more effective for irregular applications, like sparse
matrix codes, and symbolic programs such as databases, but is much 
more error prone because programmers are naturally less rigorous than compilers.  
Thread-level speculation has the potential to help simplify
parallel programming by allowing programmers and/or compilers to focus
on the *likely* case rather than the *worst* case.  Run-time support
is provided to help detect--and recover--when dependences actually exist, 
rather than requiring that they be conservatively identified apriori. This 
approach has been investigated and found promising for several classes of 
numerical applications and with varying levels of hardware support.

The novel aspects of this proposed research are two fold.  First, the
investigator intends to evaulate the applicability of these techniques to
Java applications.  To the best of my knowledge, this has not been done 
before and is an increasingly important area.  Second, they focus on 
software-centric speculation techniques. Because the overheads are higher in
software, the granularity of parallelism must be larger to compensate.
It would be a significant contribution if they could show that 
software-centric techniques perform sufficiently well to achieve good 
performance for this applications.

In summary, I think this research is interesting and relevant.  The
principle investigator has an excellent track record. I strongly
support this proposal.
Evaluator 2
PROJECT DATA
ARTES project 0005-4
Name: Techniques for Modul-Level Speculative Parallelization 
Applicants: Per Stenstrom
Overall rating: Outstanding

OVERVIEW
The project aims at finding more parallelism in realtime applications, 
such as the control software of a telephone exchange, through software-
based speculative execution. This is a continuation of a project that 
lost its funding. 

APPROACH

In order to explore more parallelism in a program, tasks that can 
not be guaranteed to be independent are started in parallel anyhow with 
the hope that no dependence will occur during their execution. If a 
dependence still occurs, it must be detected and all the side-effect 
caused by the speculative part of the execution must be undone. The 
work proposed here is  to generalize the thread-level parallelization 
and applied across consecutive modules.  The idea is that little 
dependency will be found if the modules have been developed 
using sound software practice and few global variables are used. 
The primary goal of this  project is to determine the upper limit 
achievable using ideal simulation conditions. 

SCIENTIFIC RELEVANCE OF THE PROPOSAL
This is a fairly new research area. Especially the area of
sotware-based  speculative  execution. The application of this method 
across consecutive  modules have never been  tried before. If 
successful, the methods will be applicable to a large area of existing  
software. This work has the potentials of being published in the best 
of conferences. 

The interest from the sponsor Ericsson lies in their huge investments 
in modularized  software which has been proven hard to parallelize. 

However, it is important that the overhead of the detection and rollback 
mechanism can  be kept at a minimum. The research group at Chalmers have 
reported a 10-fold speedup on  a 16-way SMP. If this approach is going 
to be successful, an even better speedup must be 
achieved, possibly through dependancy-detection supported by hardware.

MERITS AND SKILLS OF THE RESEARCH GROUP
This is one of the best research groups in Sweden. The publication
volume  and quality is paralleled by none. 

SUMMARY
A great research group that already 
Positive results have already been demonstrated during the first phase
of the project.
General and widely applicable research.
Relevant for the supporting partner Ericsson.


0005_05
Distributed Real-Time Systems with Minimal Energy Consumption: Analysis and Synthesis
Krzysztof Kuchcinski, ,
Evaluator 1
Kuchcinski		Proposal No. 5

This proposal is very relevant to ARTES and has great potential for Industrial 
use and collaboration.  The problem of real-time scheduling and energy 
minimization will become more important in the future.  The proposal fails 
to give any ideas on how they will do the scheduling.  For example, what 
heuristics will they try?  The problem seems quite limited and somewhat easy 
if it is a static scheduling problem.  In other words can't I just create some 
search heuristic or use simulated annealing to create "schedules?"  What are 
the research questions?  If there was dynamic real-time scheduling with 
minimization of energy then this would be much more challenging.  Author 
should have already chosen some real life examples to show that he thought 
about the problem.  In summary, the problem is good, but
no details on the possible solution.

Overall:  Good
Evaluator 2
Proposal: Artes 5-5

Title: Distributed Real-Time Systems with Minimal Energy Consumptions ...
Authors: Kuchcinski

Description: The authors ask for funding for the development of a
tool that constructs a feasible schedule taking while trying to
minimize energy consumption through dynamic changes of the processor
frequency. The tool work semi-automatic, i.e., an automated loop
checks the feasibility of a schedule while fine-tuning and voltage
scheduling are user-assisted.

Evaluation:

The proposal addresses an interesting problem. I am not aware of any
tool addressing a similar problem for *distributed* *heterogenous*
systems. The references seem to indicate that other work has yielded
results for uni-processors. However, the authors fail to show why
their approach is linked to distributed systems or heterogeneous
environments. They mention a project on partitioning and
scheduling at Lund University but they do not explicitly say if they
want to use their results for this study.

Some other review (by someone with a firm background on the topic)
should determine if this proposal is funded. To me, the topic sounds
good, the description lacks details but comes close to being
acceptable. Two years are realistic for one student, IMHO. (But this
proposal seems stronger than 5-21!)


Resources the authors should check out

RTAS'98 WIP: Pering/Brodersen (UC Berkeley): Energy Efficient Voltage
Scheduling for Real-Time Operating Systems

Industrial Relevance:

no company indicated, just general "believe" of relevance (which is plausible)

Further Comments:

This is not my area of specialization. My opinion has a medium to low
confidence interval :-)


0005_07
TATOO-Test and Testability of Distributed Real-Time Systems
Henrik Thane, ,
Evaluator 1
ARTES Project 0005-7
TATOO: Test And Testability of Distributed Real-time Systems
Henrik Thane
Overall evaluation: Excellent/Very Good

This proposal is for the extend support of the TATOO project for
additional 24-30 months period beginning August 2000.  The TATOO
project deals with the important fundamental problem of testing and
monitoring of distributed real-time systems (DRTS).  The project team
has developed novel methods for achieving deterministic testing,
debugging and monitoring of DRTS.

The proposed work is to implement prototype tools and evaluate them
through collaboration with an industrial partner, as well as extend
and refine the theory/notion developed for test coverage.

The result of the project will be quite useful and could serve as
foundational work in the area of systematic testing of DRTS.  The
tools and methodologies developed will be useful to industries which
build DRTS.
Evaluator 2
Tatoo		Proposal No. 7

This proposal is very different in that it is an extension to a previously
funded one.  The previous proposal is included.  The problem being worked on
is very important; there is Industrial 
support to continue funding; and a number
of results have been produced.  The problem is that without going off an
reading those referenced papers I don't know what the results are.  The new
proposal wants to continue the work, but there is not a clear statement of what
was done in phase I and what phase II will accomplish.  As far as I can tell
the basic approach tires to enumerate all possible orders, but I'm confused
as to how this reproduces the previous execution -- sure somewhere in the
complete ordering the execution that did occur exists, but how do I know
which one?  Do you test all of them?  This seems impractical.  I am probably
missing some important point, but the brevity of the description leaves me 
confused as to the proposed solution itself.

Overall:  Fair


0005_09
Support for Real-Time 3-D Graphics on Future Mobile Terminals under Energy/Area Constraints
Per Stenström, Fredrik Dahlgren,
Evaluator 1
Stenstrom - 3D Graphics		Proposal No. 9

This proposal identifies a good problem, a good approach, a novel idea and
demonstrates industry interest.  The expected results seem to be too 
conservative.  Methods and principles while important don't necessarily
produce any new results, i.e., new low power architectures.  It was not 
clear why there are no plans to actually fabricate solutions rather than just
evaluating via simulations.  For example IRAM and CRUSOE projects actually
manufactured products.  Also, while this work obviously fits well in the
ARTES program, the proposal only says that explicitly without identifying more
precisely the issues and types of solutions made necessary by real-time
concerns.

Overall:  Very Good
Evaluator 2
ARTES:   0005-9
TITLE:   Support for Real-Time 3D Graphics on Future Mobile
	 Terminals under Energy/Area Constraints
AUTHORS: P. Stenstrom, F. Dahlgren

GOAL
The proposal aims at investigating the design principles of
a general purpose single-chip architecture to provide high
and predictable performance for multimedia applications,
under the constraints of chip area and power dissipation.

COMMENTS
Providing support for 3D graphics on mobile terminals is
important for future distributed architectures and has
certainly great industrial relevance. Moreover, the proposal
is well structured, the project plan is detailed, and the
proponents are qualified to do what they promise.


EVALUATION SUMMARY  (POOR, FAIR, GOOD, VERY GOOD, EXCELLENT)
-------------------------------------------------------------
Scientific Relevance:   GOOD

Industrial Relevance:   EXCELLENT

Technical quality:	VERY GOOD

Appropriateness:	GOOD
-------------------------------------------------------------
Suggestion:             ACCEPT


0005_16
Switched Real-Time Communication for Industrial Applications
Magnus Jonsson, Bertil Svensson,
Evaluator 1
Project: ARTES project 0005-16
Title: Switched Real-Time Communication for Industrial Applications
PI: Magnus Jonsson and Bertil Svensson

Ranking: Very Good

Summary:
This project proposes to investigate real-time extensions to switched
Ethernet to provide the service guarantees needed for industrial applications.
Although the Ethernet protocol was originally designed for losses due
to collisions on the shared coax cable, higher speed implementations now
use switches which have the potential to completely eliminate packet loss.
The investigators propose to provide service guarantees for some communication
streams, while making no changes to other higher-level protocols (e.g., TCP/IP).
The latter is particularly important to allow web-based maintenance functions
to co-exist with real-time traffice.


Comments:
This work seems interesting and relevant.  Ethernet is and will continue to
be much cheaper than ATM.  Finding a way to provide quality of service
guarantees while maintaining compatibility will be a challenge; however
the results will be quite important and practical if the accomplish it.
Most switches and some adaptors provide at least limited ability
to be reprogrammed, which makes this approach quite feasible.

The prior work that that is referenced in the proposal focuses on multimedia 
streams, rather than industrial applications.  The service guarantees are 
generally less strict for multimedia, hence there is room for additional 
innovation to support industrial applications.  

I like this proposal quite a bit.
Evaluator 2
ARTES:   0005-16
TITLE:   Switched Real-time Communication for Industrial Applications
AUTHORS: M. Jonsson, B. Svensson

GOAL
The objective of this project is to analyze how methods for
supporting real-time traffic can be implemented on cheap and
simple technology, like Ethernet, for developing of industrial
embedded systems.

COMMENTS
The project addresses a very interesting issue, that could
have a strong impact on the development of future industrial
embedded systems. Reducing collisions using cheap LAN switches
which support real-time services is now feasible and worth to
investigate.
Moreover, the proposal is well written, the motivations are
clear and the intermediate goals are described in detail.
Also, the proponents have the right expertize to achieve the
expected results.


EVALUATION SUMMARY  (POOR, FAIR, GOOD, VERY GOOD, EXCELLENT)
-------------------------------------------------------------
Scientific Relevance:   GOOD

Industrial Relevance:   EXCELLENT

Technical quality:	VERY GOOD

Appropriateness:	VERY GOOD
-------------------------------------------------------------
Suggestion:             ACCEPT



0005_18
Pre-Implementation Analysis of Distributed Control Systems - PICADOR
Martin Törngren, Ola Redell, Jan Wikander
Evaluator 1
Picador - Torngren		Proposal No. 18

This proposal addresses a very difficult and important problem.  There is
actually a lot more available than referenced.  Since the expected outcome
will be a method and algorithms for pre-implementation temporal analysis,
the author should build upon previous work.  For example, can they improve
heuristics by Ramamritham, or the MARS work, or the work on using 
simulated annealing by DiNatale?  Why not combine and enhance these
heuristics rather than starting over and maybe ending up with the tool
that is not even as good as these approaches.  

Overall:  Very good. I would recommend funding this project.
Evaluator 2
ARTES:   0005-18
TITLE:   Pre-Implementation Analysis of Distributed Control
	 Systems - PICADOR
AUTHORS: M. Torngren, O. Redell, J. Wikander

GOAL
The goal of this project is to develop methods for timing
analysis of distributed real-time control systems, in order
to provide tight estimates of timing behaviour such a jitter
and control delays.

COMMENTS
The problem of providing a tight estimate of timing parameters
in a distributed application is a difficult but important
problem (both for academia and industry), because it would
allow to predict the timing behaviour of tasks, so providing
an off-line guarantee of end-to-end constraints.

The proposal is well organized, objectives are clearly stated,
milestones identified, and industrial relavance is well
described in the problem statement. Finally, proponents have
the right expertise to do the job.


EVALUATION SUMMARY  (POOR, FAIR, GOOD, VERY GOOD, EXCELLENT)
-------------------------------------------------------------
Scientific Relevance:   GOOD

Industrial Relevance:   GOOD

Technical quality:	VERY GOOD

Appropriateness:	VERY GOOD
-------------------------------------------------------------
Suggestion:             ACCEPT



0005_19
RATAD - Reliability and Timing Analysis of Distributed Systems
Hans Hansson, Christer Norström,
Evaluator 1
ARTES Project 0005-19
RATAD: Reliability and Timing Analysis of Distributed Systems 
H. Hansson and C. Norstroem
Overall evaluation: Very good

The authors propose a novel way of analysis for real-time systems that
combines low-level reliability and high-level schedulability analysis
within the same framework.  Both kinds of analysis have been employed
in separation by the real-time community for many years.  As the
authors aptly point out, schedulability analysis performed without
regard to reliability information leads to sub-optimal results during
design.  An approach that would combine the two kinds of analysis in a
workable manner is therefore highly desirable.  The topic is therefore
very relevant for the goals of the ARTES project and the current call
for proposals.

The paper provided by the authors in the appendix gives a motivating
case study for the proposed approach.  Although the analysis performed
in the case study is highly tailored to the chosen application, it is
a good starting point for the research and may lead for the
development of more general methods.

The topic appears to be a manageable project for a graduate student
under quality supervision.  The level of funding requested by the
authors is therefore adequate for the work proposed.
Evaluator 2
Proposal: Artes 5-19

Title: RTAP
Authors: Hansson ...

Description: The authors ask for funding to integrate stochastic and
deterministic models within real-time systems. Fault models from the
stochastic side should be integrated with deterministic scheduling
models to give a probabilistic approach to timing analysis, which
integrates nicely with common engineering requirements for system
safety.

Evaluation:

This work should definitely be funded. It addresses a central problem,
i.e., the lack of a probabilistic approach to timing analysis. It is
well proposed and a preliminary technical proposal shows convincingly
how the problem can be tackled, both formally and methodically. The
requested funding (1 student, 2 years) seems most appropriate.

Industrial Relevance:

While the topic itself seems relevant to industry, the proposal fails
to show any concrete industrial ties or interests for this work at the
time.

Further Comments:

You claim that distributed scheduling has been solved for real-time
seems very optimistic, given that your cited references constrain the
problem severely, i.e., for an optimal solution you do not get
preemption [Xu] and other approaches typically require a-priori
decisions about task distribution (non-optimal to begin with). A more
careful wording seems to be called for.

The tech report / submitted paper puzzles me. Fig. 1 scenario 1 claims
deadline misses but M2/M3 complete before their respective deadlines
10/20. Where are the missed deadlines? Also, you should definitely address
the problem of different execution time distributions on the
determination of confidence intervals. Next, make clear that the
phasings of interference are your random *variable*! Finally, mention
the constraints on the central limit theorem. Quantify "enough
samples" (which may be difficult but this is important in practice).


0005_20
Extension of Flexible Reliable Timing Constraints
Gerhard Fohler, ,
Evaluator 1
Fohler 		Proposal No. 20

This is an important problem that is being approached in a good manner.
Results have been produced and there is Industrial interest in the work.
A new research project has emerged from the first funded phase.  Not 
enough details were provided to understand, in depth, the new research
questions to be addressed, except for the motion of transforming reliable
timing constraints into scheduling constraints.  Overall, this is a high
quality project, should produce good results and should be funded.

Support letters were missing.

Overall: Very good.
Evaluator 2
ARTES project 0005-20
Flexible Reliable Timing Constraints (Extension)
G. Fohler
Overall Evaluation:  Very Good

An on-going ARTES-funded project concentrates on the treatment of
real-time constraints in the design of embedded real-time systems.
Its approach is based on techniques known as "end-to-end scheduling of
timing constraints," but extends to make more flexible and practical.
As such, it is very relevant for the current call for proposals.  The
project has been running for just a few months now, but several
important results have been achieved already.  The interest of the
industrial partners in the results of the project promises to increase
the scope of the project even further.

The scope of project has clearly exceeded the current level of funding
by ARTES.  One additional graduate student, asked by the proposed
extension, is fully justified by the current state of the project.
The fact that there is a student with experience in the project, who
can start working immediately, makes the extension even more worthy.


0005_22
Real-Time Response and Control of Autonomous Agents
Nancy E. Reed, ,
Evaluator 1
Reed   Real-Time Response and Control...  Proposal 22

This proposal is about real-time AI.  The problem with real-time AI
work is that it ignores most of the real-time problems. While
one might be able to use anytime algorithms conceptually, without
linking the high level planning etc to real system level code,
hardware and sensors, very little can truly be said about the
real-time properties of the system. What does efficient mean in
your context?  Does it execute in no more than milliseconds?
does it meet a deadline? or is it just polynomial? Is it an
anytime algorithm? Your update to the Pirjanian algorithm seems
like a straightforward use of the anytime idea for this algorithm.
Why is this a significant research/novel idea? What experiments will
you run? How will you run them? Will the results be used on some
real hardware or in simulation? 

Overall:  Fair.
Evaluator 2
ARTES:   0005-22
TITLE:   Real-Time Response and Control of Autonomous Agents
AUTHORS: N. Reed

GOAL
This project proposes to investigate solutions to the problem
of real-time response and control of autonoumous agents, that
is systems that are embedded in an environment with sensors
for obtaining data and effectors for performing actions.
The project aims at addressing not only the problem of action
selection within timing constraints, but also other related
objectives, such as execution monitoring and long term
planning.

COMMENTS
The proposal addresses an interesting scientific problem, whose
solution has important implications in many application domains,
such as internet searching, economic markets, environment
monitoring, and robotics. Hence, the proposed research has also
a high industrial relevance.
The proposal is well organized, motivations and goals are
clearly stated, milestones are identified.
The only concern of this reviewer is that, although the
proponents have a good expertize in the field of artificial
intelligence and autonomous agents, they never worked on
real-time systems.



EVALUATION SUMMARY  (POOR, FAIR, GOOD, VERY GOOD, EXCELLENT)
-------------------------------------------------------------
Scientific Relevance:   EXCELLENT

Industrial Relevance:   EXCELLENT

Technical quality:	FAIR/GOOD

Appropriateness:	VERY GOOD
-------------------------------------------------------------
Suggestion:             WEAK ACCEPT



0005_23
Embedded Databases for Embedded Real-Time Systems
Jörgen Hansson, Christer Norström,
Evaluator 1
Jorgen Hansson - Embedded Databases	Proposal No. 23

This proposal addresses a new important, and emerging area.  The project
seems very ambitious.  While the authors have Industrial partners it is
not clear how they will work with them.  Very difficult issues include
the tradeoff-analyses and the composition rules.  What are the approaches
in these 2 areas?  Will combining components provide good enough performance?
Where and where not has this approach worked?  In general these are very
difficult non-functional issues involved with composing components.  

Overall: Very good.
Evaluator 2
ARTES:   0005-23
TITLE:   Embedded Dadabases for Embedded Real-TIme Systems
AUTHORS: J. Hansson, C. Norstrom 

GOAL
In this project, the proponents want to provide a database
system with minimum memory requirements and necessary
functionality, to be used in industrial embedded applications.
In particular, the project focuses on design tools and
methodologies for developing embedded databases with real-time
services.

COMMENTS
The problem is well formulated and the proposal is technically
sound. The expected results have high industrial relevance
due to the large application domains where embedded databases
can be used.
The project plan is clear and the two academic partners have
an excellent complementary expertize in real-time databases
and real-time systems to investigate the proposed issues.


EVALUATION SUMMARY  (POOR, FAIR, GOOD, VERY GOOD, EXCELLENT)
-------------------------------------------------------------
Scientific Relevance:   VERY GOOD

Industrial Relevance:   GOOD

Technical quality:	EXCELLENT

Appropriateness:	VERY GOOD
-------------------------------------------------------------
Suggestion:             ACCEPT
  ---------------------line----------------------------
  Strategic Research