Minutes of the 4 th session of VMAD IWG
Date : October 16-17, 2019
Venue : Novotel Ottawa , Canada
Participants : Canada / TC, China/CATARC, EC, France, Japan (MLIT/NTSEL/JASIC), Korea/KATRI,
the Netherlands/RDW, Sweden/SWE TA, UK (VCA/TS Catapult), US/NHTSA, AAPC, CIECA, CLEPA, EGEA, OICA, SAE
Observer: Intel (Mobileye), Thatcham, Uber (Total: 50 participants)
Day 1: October 16 (Wednesday) 10:00 – 17:00
Agenda Item 1: Adoption of the agenda (VMAD-04-02-rev.2)
Welcome remarks were made by representative of Canada, Co-Chair of VMAD.
Participants i ntroduced themselves because some new faces were seen.
Provisional agenda was approved without any modifications.
Agenda item 2. Adoption of the report of the 2 nd VMAD IWG meeting (VMAD-04-01-rev.1)
T he report was approved with modification to the second paragraph of agenda item 5. (5) proposed by OICA. Refer to VMAD-04-01-rev.2.
Agenda item 3. Report on the 4 th GRVA
Co-Chair/Japan briefed on the highlights of the 4 th GRVA session.
As for ALKS, would the following assignment to ACSF IWG and VMAD IWG correct? (OICA)
-ACSF would consider requirements and classical test methods.
-VMAD would consider new test methods.
It was agreed that both IWGs would continue their current activities. (Co-Chair/J)
ACSF IWG was formed to amend UN R79 . However, it w as about to go beyond the scope. The Framework Document clarified that ACSF IWG would only deal with an UN regulation on ALKS functional requirement. (USA)
Agenda item 4. Clarification of VMAD and FRAV synergy/ boundaries (FRAV-01-15-rev.1)
FRAV Secretary (AAPC) explained FRAV-01-15-rev.1.
FRAV and VMAD IWGs should cooperate each other with requirements/ assessment methodologies from “high level” to “details”. It is important to go ahead with the task anyway although there might be some grey (fuzzy) “detailed” areas where it is not clear as to which IWG should take the lead. (USA)
Which IWG would consider traffic scenarios? (France)
Subgroup of VMAD (SG 1a) already started consideration of traffic scenarios. However, cooperation between FRAV and VMAD IWGs would be necessary on traffic scenario issue from its early stage because “how to fix the boundary between preventable and unpreventable traffic scenarios” is extremely important. (EC)
What would be the outcomes of both Subgroups? (Japan/NTSEL)
They are just as the “V” shaped figure at slide 4 of FRAV-01-15-rev.1 illustrates. Flexible approach would be envisioned. ( C o-Chair/The Netherlands)
VMAD IWG agreed to the following FRAV-VMAD responsibilities shown at slide 3 of FRAV-01-15-rev.1;
- Nece ssary for both IWGs to start from “high level” and work down towards necessary level of specificity
-FRAV will start focusing on their efforts on defining what is considered a “safe” automated vehicle
- VMAD will start focusing on their efforts on defining how to assess if a vehicle is “safe”
ACSF IWG activities are limited to the scope of the 1958 Agreement. Both FRAV and VMD IWGs should not be bound by the outcome of ACSF IWG activities. (USA)
Deliverables of FRAV and VMAD IWGs in next March are fixed. (EC)
Agenda item 5. Report from Subgroups
(1) Subgroup 1a: Traffic Scenarios
Leader of SG 1a (Japan/MLIT) illustrated VMAD-04-03 “SG 1a: Outcome, Q&A”.
It would not be acceptable to leave unpreventable scenarios untouched. ADS should react to unpreventable scenarios. These reactions of ADS should be assessed properly. (France)
Remarks on the boundary between preventable and unpreventable scenarios (USA)
i) No clear boundary
ii) The boundary shifts with technological advancement
iii) Political decision would be necessary to fix the boundary
It is necessary to continue discussion on this important issue. (EC)
As the first step it would be the only way to fix the boundary as e.g., “skilled driver” because there are no automated driving vehicles in the market now. The boundary should be reviewed periodically in accordance with technological advancement of ADV in the market. (Japan/NTSEL)
There might be some crucial unforeseeable scenarios. (SAE/ Waymo)
The term “accident-preventable” might be more precise. (EC)
---------------------------------------------------- Lunch break----------------------------------------------------------------------
Where should we discuss on scenario database? (CLEPA/NVIDI A )
The scenario database would be necessary in the long run. I t could not be made in the short run . (Japan/NTSEL)
It would be necessary to decide the course of action by Mach, 2020. (EC)
As for Question 1-2, the traffic scenario could be used to validate both the safety of the intended functionality and functional safety. (CLEPA/NVIDIA)
It would be desirable but it would be difficult to validate all by the sole simulation. (EC)
Japan/JASIC explained VMAD-04-04 “Safety Criteria Study on Innovative Safety Validation Methods of Automated Driving System”.
Automated driving system would be expected to perform better than skilled human driver in France. It would be FRAV that should decide the performance level of automated driving system. (France)
It could not be said that cognitive performance of system would surpass that of human. How to handle issues relating to “preventable vs. unpreventable”? System performance level could be skilled human driver plus ADAS. (EC)
The definition of “skilled human driver” would be necessary. (Canada)
Data on skilled human driver in different countries are available. They are of some help. (CIECA)
Careful attention should be paid to the rate of “accidents related to human factors” to be 97%. Cases of safe driving by human drivers are not included in these data. (EC)
(2) Subgroup 2a: Audit/ Virtual testing/ In-use data reporting
Leader of SG 2a (EC) explained VMAD-04-05 “SG 2a: Outcome, Q&A”
As to the following Q&A, what does 3 rd party mean? (France)
Q: Who should conduct the audit/assessment?
A: Audit and assessment to be conducted by a qualified independent 3rd party.
Technical Service, etc. (EC)
In-use data are also beneficial to insurers. (Co-Chair/N)
How to link in-use data with system performance validation? (EC)
DSSAD could be used. (OICA)
(3) Subgroup 2b: Test track/ Real-world testing
Leader of SG 2b (CIECA) explained VMAD-04-06 “SG 2b: Progress of work”.
What are the differences between track testing and real-world testing? (EC)
They are to be discussed by SG 2b. (CIECA)
Agenda item 6. Validation of new test and assessment methods when applied to ALKS
Co-Chair/J explained the state of play.
Day 2: October 17 (Thursday) 9:00 – 15:00
Small drafting group including Leaders of SG 1a, 2a, and 2b, and chair of CEL/TF will be organized to draft informal document to address new test and assessment methods (including CEL) when applied to ALKS. (Co-Chair/C)
CEL is somewhat similar to audit. Therefore, CEL should be handled by SG 2a. There is no use creating another organizational layer. (EC)
VMAD IWG and its subgroups should focus its activities on new test and assessment methods. (USA)
Small drafting group consisting of the existing organization will be formed to meet the deadline of March, 2020 stipulated in the framework document. (Co-Chair/J)
Can OICA participate in the small drafting group? (OICA)
Agenda item 7. Autonomous driving safety validation: Proposals from the French Eco-system
France explained VMAD-04-13 “New validation approaches for automated driving safety”.
Who would be responsible for scenario management, authorities or manufactures? (OICA)
Yet to be decided. Manufacturers might be responsible at the outset. (France)
Is this guidelines or regulations? (Co-Chair/C)
After introduced as international regulations, they will be incorporated into domestic legislation. (France)
Agenda item 8. Defining Safe Automated Driving
Thatcham Research explained VMAD-04-09-rev.1 “Defining Safe Automated Driving”.
Would it be possible for 3 rd party to have access to data? (OICA)
Such data belong to each individual who owns the car. (Thatcham Research)
Have validation requirements been applied? (Co-Chair/N)
Some of validation requirements have been already applied although application items are different between ADS and ADAS. (Thatcham Research)
Agenda item 9. Update on the UK scenario database project
UK/Catapult explained VMAD-04-11 “Update on the UK scenario database project”.
UK scenario database could contribute to test scenario under development by SG 1a (UK/Catapult)
Agenda item 10. The status of China’s real world testing for AV
China/CATARC explained VMAD-04-12 “The status of China’s real world testing for AV”.
What are pass/fail criteria? (CLEPA/NDIVIA)
To be determined in the future. (China/CATARC)
Would the influence by other road users be assessed? (Co-Chair/N)
Not sure. (China/CATARC)
Agenda item 12. Building a framework for self-driving safety
Uber explained VMAD-04-10 “Building a framework for self-driving safety”.
Which tools (track testing or simulation) would be used? (CLEPA/NVIDIA)
Various tools supplementing one another are used. (Uber)
Do you have any specific recommendations to VMAD IWG? (Co-Chair/N)
Nothing in particular. (Uber)
Agenda item 15. AOB
If possible, future FRAV and VMAD IWG meetings will be held in a row in the same week in this order.
Action items and timeline until the next VMAD IWG meeting to be held in January, 2020 in Tokyo are shown in VMAD-04-14-rev.1.