Page tree

 

 

VMAD

Subgroup 2b: Case study ALKS

Test proposal ALKS test track and real world testing

(Based on document SG2b “General overview test approach”)

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Brussels, 29 November 2019

Definition on Automated Lane Keeping Systems (ALKS)

As described in the general approach document of SG 2b concerning testing automated vehicles on a test track and real world testing, as a next step a case study will be designed regarding ALKS. At the moment there is a variety in different ALKS systems (a.o. operable from/to a certain speed of the vehicle). In spite of the heterogeneity of the ALKS systems the basic principle for testing these systems is that they should perform safely under all traffic situations and this will be leading for the tests on and test track and also in real world testing.

 

Starting from traffic scenarios

As mentioned before, traffic scenarios will be the basis when starting tests on a test track. Between subgroup 1a and 2b there is co-operation for to the development and exchange of relevant (critical) scenarios. At the moment of writing this case study not all scenarios are available yet. Therefore some examples, designed by the experts of Japan will be used in this case study to clarify and describe the principles of testing on a test track and in real world.

 

Concept for testing on test track and real world

The most important aim of this case study is to design a general description and logical steps to be taken for testing, to demonstrate safe traffic behaviour on public roads. Once the principles of this proposal have been discussed (In Tokyo, January 2020), elaborated and accepted, further development will be  done by SG 2b. A (final) prototype for the test methods will be delivered spring 2020 to be presented to GRVA. It should be noted that after acceptance a validation process for the test methods will be a next step.

 

In short, the following steps should be developed:

  1. A minimum set of tests to be defined for ALKS;
  2. Those tests which are proposed from the audit/documentation/simulation/virtual testing which are still part or should be of the minimum set;
  3. Those tests which seem reasonable from 1 and 2 and/or (in case of real world) from track testing.

 

 

Approach test track

After passing phase 1 (virtual testing) either performed by the (car)manufacturer or an independent, third party (see also proposals of subgroups 1a and 2a), the car will be tested in the next step (step phase 2) on a test track. Offering the car will be accompanied by a test report in which performing of the car (on ALKS) is described. The report could contain the performance of the car in the virtual surrounding regarding the operational performance, limits and conditions. This will be the input for testing on the test track.

The virtual world should be a ‘digital twin’ of the roads where the system is going to be used.

 

Safety envelope

First of all a system should do for what it is designed. Meanwhile the car (system) should be capable of maintaining a safety envelope in the direct surroundings of the car. This means, dependant of speed, specific (traffic) situations there should be a safe space ( “cushion”) in the direct surroundings of the car in relation to other road users.

 

During track- (and also in real world) testing, the car will be in general tested whether it is capable to maintain this envelope under all (test) circumstances.

 

Selection of (critical) traffic scenarios

First of all the car will be (generally) tested using several scenarios. Some examples (to be elaborated):

 

Keep your lane

Source image:

 

  • Is the vehicle capable of maintaining the lane while using several linings?
  • At what speeds is the system working and how?
  • Is the vehicle capable of maintaining the safety envelope?

 

 

ALKS and road geometry

Source image:

 

  • Is the ALKS system capable recognizing different road structures/linings etc. and keep its lane?
  • Can the vehicle handle, while keeping its lane, being merged (several situations).
  • How is the system performance of the vehicle while changing lane, merging, overtaking?
  • Does the system recognize curves, ramps and is it capable to deal with (emergency) lane change manoeuvres?
  • Interaction system – driver? / Interaction other road users?
  • Does the ALKS recognize the boundaries of the ODD and does is have a clear and acceptable handover to the driver

 

 

 

ALKS system “under stress”

Elements can be added to the basic scenarios, a.o.:

 

  • No or very poor lane marks
  • Speed differences of the car
  • Daylight / darkness
  • Variation on surfaces: water, oil, mud, garbage
  • Interference of cyclists, pedestrians, men at work
  • Closure of lanes (road works)
  • Presence of emergency vehicles
  • Human machine interface (HMI)
  • Performance in tunnels
  • How does the system react when it is (sensors) are failing (information driver / blocking of functionality)
  • Interaction of ALKS with other (automated) systems (systems shouldn’t block others)

 

Elaboration of test design test track

The scenarios and traffic situations are a starting point and should be further elaborated by SG 2b (and SG 2a) to design a complete test battery applicable for the test track.

The test track should have the same infrastructure elements that the vehicle will encounter in the real ODD.

 

Test protocol

Test criteria are based on:

  • Safety  - awareness of hazards, ability to modify behavior in order to prevent accidents happening
  • Coverage of the traffic situations (abrupt changes)
  • Coverage of the environments (urban area, outside urban area, mixed traffic, more controlled environment)
  • Coverage of circumstances (demanding conditions; weather, congestion)
  • Human intervention (need)
  • Fluent and smooth driving

 

 

Example for a test protocol:

 

While testing ALKS on a test track or in real world, an assessment protocol is needed. The example used here is to show how criteria can be set and connected to (critical) values. On the left side are some general traffic situations while the car is driving (driving on straight and curvy roads, merging, exiting, etc.). These general road situations can be connected to “behaviour” of the car in that specific situation. Values (criteria) can be added to the matrix. The matrix is just meant as an example. This needs to be further elaborated by the members (a.o.) of subgroup 2b. It also needs to be noted that, once the test method(s) an d the matrix has been developed, the test method needs validation before being officially used.

 

 

 

 

 

Assessment matrix (Example)

 

 

 

 

Keeping proper place within lane

 

 

 

 

 

 

Ability to deal with several linings

 

 

score

 

 

 

 

Interaction of system with driver

 

 

 

total

 

 

 

 

Maintaining safety envelope …..

Behaviour on the road

 

6

 

 

 

 

 

Driving on straight and curvy roads

 

Encountering/being overtaken

(3)

 

 

 

 

 

 

(3)

 

 

 

 

 

 

Changing of direction

 

 

 

 

 

 

 

 

Merging / exiting

(3)

 

6

 

 

 

 

 

 

Overtaking / passing

(3)

 

 

 

 

 

 

 

Lane changing/lateral movements

(4)

4

 

 

 

 

 

 

………

 

 

 

 

 

 

 

 

………

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Total score

…..

 

 

 

 

 

 

 

 

 

Real world testing

Once the vehicle has passed the stage of track testing, the next stage will follow: a test drive in the “real world traffic”. The difference with the test track is there will be no pre-set conditions. During the real world test however the vehicle will drive during a certain amount of time in traffic on public roads. This means specific situations might occur (with other road users) and the vehicle (in this case ALKS) will be tested using the same matrix as presented in the previous chapter. But possibly filled with another set of criteria; a more “general approach” following in traffic testing / and or other specific tests added.

 

A specific test route (in which for instance several road types are inserted) might be defined. Also this part will have to be elaborated by subgroup 2b and also validated In a later stage.

 

Finally it should be defined what should be reported on the basis of the tests performed and how these reports should look like:

 

  • Which tests have been performed
  • Under which circumstances
  • What are the results
  • Conclusions