Getting the future right – Artificial intelligence and fundamental rights

AI and machine learning are different concepts. AI is an umbrella term."
(Private company, Estonia)

"What you see now is that everyone doing something with machine learning is labelling this as ‘AI’."
(Public administration, Netherlands)

"[AI] is mostly used to save time […] when you have to go through a lot of material."
(Public administration, Netherlands)

"The most important is to deal with cases more efficiently. It’s about making use of your workforce, the people who handle cases, as effectively as possible."
(Public administration, Netherlands)

"[Our use of AI] does not impact [human rights] in any way. In terms of the decision process, it does not matter whether the decision is made by machine or a human."
(Interviewee working for public administration, Estonia)

"Internally we can explain the decisions of the machine learning models and we have several means to do that."
(Private sector, Estonia)

"If the systems do not have black boxes of information or processes, we already take a step forward in the defence of human rights."
(Public administration, Spain)

"We are strongly attached to the idea that AI has to be explainable."
(Public administration, France)

"Once all the rights related to data protection are ensured, I do not see how human rights are of relevance here."
(Private company, Spain)

"We did not touch the topic because we assume that there are no human rights issues involved: all the activities are within the legal framework, all the activities are compliant with data protection and good practices, and therefore we assume that there are no human rights issues related to these systems."
(Public administration, Spain)

"I do not think that we should regulate specific technology like AI. It is sufficient to have general principles and technology-neutral rules."
(Private sector, Estonia)

"We were a little anxious when the GDPR was implemented, but in the end it meant managing datasets and access rights […] It is a good reminder that not everything can be or should be done."
(Public administration, Finland)

"Actually, I’m concerned that the GDPR might hinder AI research. I’m afraid that some large databases that we have used previously cannot be used for our research anymore."
(Private company, Netherlands)

"There is the GDPR but it does not give you specific rules. It gives principles but it comes down to ethical issues and interpretation."
(Private company, Estonia)

"There is a huge tension surrounding the GDPR. So we want to do well, but might in fact be worse off, because interpretation of the data then turns out to be impossible."
(Public administration, Netherlands)

"If we had to explain the model, we wouldn’t be able to. The model is statistical and not very explainable."
(Public administration, France)

"The number of the complaints about data use is miniscule, rather people may have asked to delete some information about them."
(Private company, Estonia)

"Yes, we assess the legality of personal data protection and the conformity with their specific legal acts."
(Public administration, Estonia)

 

"If you want the machine not to discriminate on the basis of sex, do not put the variable of sex, as easy as that, or make the examples symmetrical if you notice that sex has certain relevance."
(Public administration, Spain)

"For discrimination, it’s complicated because some diseases are more present in certain ethnic groups. Predictions take into account the sexual, ethnic, genetic character. But it is not discriminatory or a violation of human rights."
(Private sector, France)

"We try to look into the future. We will automate more and more."
(Private company, Estonia)

"The next steps are related to transparency and open data: that is to say, publish not only information in pdf, but also information in reusable formatting so that it could be reused internally and by the private sector."
(Public administration, Spain)

"AI is a great thing but we must learn to use it."
(Private company, Spain)

"When testing the system, we did not really look at the legal aspects, we looked at whether the system is profitable."
(Private company, Estonia)

"There is a risk of having too much trust in the machine."
(Public administration, France)

"Yes, we assess the legality of personal data protection and the conformity with their specific legal acts."
(Public administration, Estonia)

When introducing new policies and adopting new legislation on AI, the EU legislator and the Member States, acting within the scope of EU law, must ensure that respect for the full spectrum of fundamental rights, as enshrined in the Charter and the EU Treaties, is taken into account. Specific fundamental rights safeguards need to accompany relevant policies and laws.

In doing so, the EU and its Member States should rely on robust evidence concerning AI’s impact on fundamental rights to ensure that any restrictions of certain fundamental rights respect the principles of necessity and proportionality.

Relevant safeguards need to be provided for by law to effectively protect against arbitrary interference with fundamental rights and to give legal certainty to both AI developers and users. Voluntary schemes for observing and safeguarding fundamental rights in the development and use of AI can further help mitigate rights violations. In line with the minimum requirements of legal clarity – as a basic principle of the rule of law and a prerequisite for securing fundamental rights – the legislator has to take due care when defining the scope of any such AI law.

Given the variety of technology subsumed under the term AI and the lack of knowledge about the full scope of its potential fundamental rights impact, the legal definition of AI-related terms might need to be assessed on a regular basis.

The EU legislator should consider making mandatory impact assessments that cover the full spectrum of fundamental rights. These should cover the private and public sectors, and be applied before any AI-system is used. The impact assessments should take into account the varying nature and scope of AI technologies, including the level of automation and complexity, as well as the potential harm. They should include basic screening requirements that can also serve to raise awareness of potential fundamental rights implications.

Impact assessments should draw on established good practice from other fields and be regularly repeated during deployment, where appropriate. These assessments should be conducted in a transparent manner. Their outcomes and recommendations should be in the public domain, to the extent possible. To aid the impact assessment process, companies and public administration should be required to collect the information needed for thoroughly assessing the potential fundamental rights impact.

The EU and Member States should consider targeted actions to support those developing, using or planning to use AI systems, to ensure effective compliance with their fundamental rights impact assessment obligations. Such actions could include funding, guidelines, training or awareness raising. They should particularly – but not exclusively – target the private sector.

The EU and Member States should consider using existing tools, such as checklists or self-evaluation tools, developed at European and international level. These include those developed by the EU High-Level Group on Artificial Intelligence.

The EU and Member States should ensure that effective accountability systems are in place to monitor and, where needed, effectively address any negative impact of AI systems on fundamental rights. They should consider, in addition to fundamental rights impact assessments (see FRA opinion 2), introducing specific safeguards to ensure that the accountability regime is effective. This could include a legal requirement to make available enough information to allow for an assessment of the fundamental rights impact of AI systems. This would enable external monitoring and human rights oversight by competent bodies. The EU and Member States should also make better use of existing oversight expert structures to protect fundamental rights when using AI. These include data protection authorities, equality bodies, national human rights institutions, ombuds institutions and consumer protection bodies. Additional resources should be earmarked to establish effective accountability systems by ‘upskilling’ and diversifying staff working for oversight bodies. This would allow them to deal with complex issues linked to developing and using AI. Similarly, the appropriate bodies should be equipped with sufficient resources, powers and – importantly – expertise to prevent and assess fundamental rights violations and effectively support those whose fundamental rights are affected by AI. Facilitating cooperation between appropriate bodies at national and European level can help share expertise and experience. Engaging with other actors with relevant expertise – such as specialist civil society organisations – can also help. When implementing such actions at national level, Member States should consider using available EU funding mechanisms.

EU Member States should consider encouraging companies and public administration to assess any potentially discriminatory outcomes when using AI systems.

The European Commission and Member States should consider providing funding for targeted research on potentially discriminatory impacts of the use of AI and algorithms. Such research would benefit from the adaptation of established research methodologies, from the social sciences, that are employed to identify potential discrimination in different areas – ranging from recruitment to customer profiling.

Building on the results of such research, guidance and tools to support those using AI to detect possible discriminatory outcomes should be developed.

The European Data Protection Board (EDPB) and the European Data Protection Supervisor (EDPS) should consider providing further guidance and support to effectively implement GDPR provisions that directly apply to the use of AI for safeguarding fundamental rights, in particular as regards the meaning of personal data and its use in AI, including in AI training datasets.

There is a high level of uncertainty concerning the meaning of automated decision making and the right to human review linked to the use of AI and automated decision making. Thus, the EDPB and the EDPS should also consider further clarifying the concepts of ‘automated decision making’ and ‘human review’, where they are mentioned in EU law.

In addition, national data protection bodies should provide practical guidance on how data protection provisions apply to the use of AI. Such guidance could include recommendations and checklists, based on concrete use cases of AI, to support compliance with data protection provisions.

The EU legislator and Member States should ensure effective access to justice for individuals in cases involving AI-based decisions.

To ensure that available remedies are accessible in practice, the EU legislator and Member States could consider introducing a legal duty for public administration and private companies using AI systems to provide those seeking redress information about the operation of their AI systems. This includes information on how these AI systems arrive at automated decisions. This obligation would help achieve equality of arms in cases of individuals seeking justice. It would also support the effectiveness of external monitoring and human rights oversight of AI systems (see FRA opinion 3).

In view of the difficulty of explaining complex AI systems, the EU, jointly with the Member States, should consider developing guidelines to support transparency efforts in this area. In so doing, they should draw on the expertise of national human rights bodies and civil society organisations active in this field.