AI governance in the organization: roles and responsibilities

Who approves a new AI tool, and who is accountable for the data and the outputs? Roles and decision rights for governing AI use in the organization.

Topic
AI
Author
ARLITECH expert team
Published

AI tools find their way into an organization by several routes. IT procures some of them, a department subscribes to others out of its own budget, and an employee starts using still others with a personal account. An update to existing business software can also bring a new AI feature that nobody decided on separately. If nobody has been designated to say yes or no to these, use begins without approval and without a record.

AI governance sets out who decides whether an AI tool may be used, who is accountable for the data that goes in and the outputs that come back, and who checks that the rules actually apply in day-to-day work. This article goes through the roles and the decision rights. Technical enforcement of the rules, including filtering and logging, is covered in the article on controlling the use of AI services.

Why AI use needs an owner

The risk of an AI tool arises in several places at once. Data the user types in or uploads can end up with an external provider. The output can be wrong while sounding confident. An AI agent or an integration can also carry out actions in other systems on the user’s behalf. These questions belong to different disciplines: data protection, IT security, legal, and the owner of the process the tool supports.

If each area looks only at its own slice, the decision falls apart. The security team signs off on the service because the connection is encrypted and single sign-on works, while nobody has asked what customer data will go into it. Legal reviews the contract without knowing that the tool’s outputs will go straight to customers. Governance makes sure that every use of AI has a named owner who is responsible for the whole picture and knows whom to bring in.

International frameworks start from the same place. Of the four functions in the NIST AI Risk Management Framework (AI RMF 1.0, NIST AI 100-1, 2023), Govern frames the other three, Map, Measure, and Manage, and it is where roles, responsibilities, and accountability belong. ISO/IEC 42001:2023, the standard that sets out the requirements for an AI management system, expects top management to assign responsibility and authority for the relevant roles and to communicate them within the organization. Neither is mandatory, but both give you a workable structure for setting up your own arrangements.

Decision rights: approval, prohibition, exception

Most questions about AI use end in one of three kinds of decision. It helps to name them separately, because each has a different decision-maker and a different turnaround time.

Approval

A new tool, or a new purpose for a tool already in use, is approved with conditions: which account (corporate or personal), which data categories, and which group. The approval applies to a purpose of use, and to the tool only as part of that. The same writing assistant can be fine for drafting marketing copy and still not be acceptable for correspondence that contains customer data.

Prohibition

There are grounds for a prohibition if the tool’s terms of use are unacceptable (for example, the provider uses input data to train models and this cannot be turned off), if the purpose is prohibited by law, or if the organization cannot provide the oversight the use would need. What keeps a prohibition enforceable, and how unreported use comes to light, is covered in the article on shadow AI.

Exception

An exception is a departure from a rule, limited in time and scope. A team may, for example, use a tool that is not generally approved for the duration of a pilot project. An exception has a requester, a justification, compensating conditions (only anonymized data may go into the tool, for instance), an expiry date, and an approver who takes on the residual risk, that is, who makes the risk acceptance decision. On the expiry date the exception lapses unless the approver extends it with a new decision.

Decisions are best delegated according to risk. IT can approve a low-risk request that works with public data against a fixed checklist, while cases involving personal data, customer data, or decisions about people go through the full review and need management approval. Turnaround time matters as well: if a request goes unanswered for weeks, the employee will very likely find an unapproved workaround. Every decision has to be recorded in writing, with the decision-maker’s name, the justification, and the conditions.

Roles: management, data owner, security, legal, user

Role names vary from one organization to another, and at a smaller company one person often holds several roles. The responsibilities stay the same.

Management

Management sets how much risk the organization is willing to take on with AI use, adopts the AI usage policy, and provides the resources to carry it out. It appoints the coordinator who handles requests and the inventory, and a business owner for each use, typically the head of the department whose process the tool supports. Management decides on high-risk uses and significant exceptions.

Data owner

The data owner decides which of the data categories in their care may go into a given AI tool, and on what terms. The existing data classification is the basis for this: each classification level can be mapped to whether it may go into a public service, only into a tool used with a corporate account, or nowhere. If an AI tool works from internal document repositories, the data owner also decides which repositories it may draw on and who may see the answers it produces from them. The technical side of this is covered in the article on access control in RAG systems.

IT and security

IT and security are responsible for the technical review of the tool: authentication and multi-factor sign-in, where the data is stored and how long it is retained, administrator settings, logging, and integrations with other systems. For a tool that works as an agent, a separate question is what permissions it runs with and what it can do without human approval, which the article on AI agent permissions covers. This area enforces the decisions in practice (allowing, blocking, provisioning accounts) and also handles AI-related security incidents.

Legal and data protection

Legal reviews the contract and the terms of use: whether the provider uses the input data, how long it keeps it, where it stores it, who its subcontractors are, and who holds the rights to the output. The business owner then decides whether to accept the terms, or management if the risk is high. When personal data is processed, the GDPR requires considering a data protection impact assessment (Article 35), for which the controller must seek the advice of the data protection officer, if one has been designated. It is also legal’s job to determine the organization’s role under the AI Act and to flag when a planned use comes with an obligation to inform employees under labor law.

User

Users work only with approved tools, under the conditions of the approval. They check the output before using it, especially when it goes to a customer, an authority, or another decision-maker, and they label AI-generated content where the policy requires it. If they need a new tool or a new purpose, they request it through the designated channel, and they report it when a tool gives wrong or questionable results, or when sensitive data has gone into it by mistake.

The table below shows who decides and who has to be involved in the typical questions. For a finer breakdown, you can use the four roles of a RACI matrix (responsible, accountable, consulted, informed).

QuestionDecidesInvolve
Adopting and amending the AI usage policyManagementSecurity, legal and data protection, data owners
Approving a new tool or a new purpose of useThe business owner of the use, management if the risk is highSecurity, legal and data protection, data owner
Which data categories may go into the toolData ownerData protection, security
Configuring authentication, logging, and integrationsIT and securityBusiness owner
Accepting the contract and terms of useBusiness owner, management if the risk is highLegal, procurement, data protection
Granting an exceptionThe manager who takes on the residual riskSecurity, legal
Using a specific output in your workUserBusiness owner (who sets how the output is checked)
Practical tip: put a person’s name in every row of the table, because a department name is not enough. If the responsible person leaves or changes jobs, the uses listed under them in the inventory have to be handed over, or they keep running with no owner.

AI inventory and risk classification

Decisions are made per purpose of use, so the AI inventory should also have one row per use: a single tool name can have several rows, each with its own owner and conditions. From a governance point of view, what matters is that each row shows who decided (role and name), on what conditions, who the business owner is, and which integrations and permissions belong to the use. The general inventory fields are listed in the article on EU AI Act basics.

The internal risk classification can be based on a few criteria:

  • Data sensitivity. Whether public material, internal documents, personal data, trade secrets, or strictly confidential data go into the tool.
  • Impact of the output. Whether it only supports the user’s own work, goes outside the organization, or feeds into decisions about people.
  • Autonomy. Whether it makes suggestions that a person approves, or carries out actions in other systems itself.
  • External visibility. Whether it is an internal tool, or customers and partners deal with it directly.

The internal level and the category under the AI Act are two separate fields. A tool that summarizes internal documents may be minimal risk under the Act and still rate high internally if it processes strictly confidential data. It also works the other way around: a tool that works with public data can become high-risk under the Act if it is used to evaluate employee performance. Different people decide the two levels. The business owner sets the internal level based on the coordinator’s proposal, and legal sets the category under the Act, because that determines which obligations the organization has to meet.

How this ties in with deployer obligations under the EU AI Act

Regulation (EU) 2024/1689 uses the term deployer for an organization that uses an AI system under its own authority in the course of its professional activity. The categories and the obligations are explained in the article on EU AI Act basics. The question here is which obligation lands with which role.

  • AI literacy (Article 4). Applies to every deployer, whatever the risk category. Management is accountable for it, HR typically organizes the training, and security and legal supply the content for each role.
  • Transparency (Article 50(4)). If the organization publishes a deepfake, or publishes AI-generated text to inform the public on matters of public interest, it has to disclose that the content was artificially generated. For text, this is not required if a person has reviewed it and someone holds editorial responsibility for it. The labeling rule is a shared task for communications and legal.

The further obligations in Article 26 apply only to deployers of high-risk systems. The basics article covers what they involve; responsibility for them can be split across the roles like this:

  • Human oversight. Here the regulation explicitly expects roles to be assigned: the business owner designates the person who carries out oversight, and management gives that person the necessary authority.
  • Use in line with the instructions, monitoring of operation, and reporting. The business owner.
  • Keeping the logs. IT.
  • Informing employees before the system is introduced. HR and legal.
  • Data protection impact assessment. The GDPR assessment mentioned above has to draw on the information supplied by the provider. This falls to legal and data protection.

Under Article 25, a deployer can become a provider, for example if it puts its own name or trademark on a high-risk system that has already been placed on the market or put into service, or if it starts using a system for a purpose that makes it high-risk. The approval process has to catch this, which is why legal sees every request that would put an already approved tool to a new purpose.

Regular review

A decision is valid as long as its conditions hold, and an AI tool keeps changing after it has been approved. An unscheduled review is worth starting if:

  • the provider changes its terms of use or its data processing terms;
  • the provider moves to a new model, or the tool gains a new capability, such as file upload, web search, or carrying out actions in other systems;
  • there has been an incident or a near miss, for example a wrong output reached a customer, or sensitive data went into a tool not meant for it;
  • the applicable law changes, or the authority issues new guidance.

The scheduled review goes through the entire inventory. Is the tool still in use, or can the accounts and subscriptions be closed? Have exceptions expired, and if so, has the use stopped or is a new decision needed? Is the designated owner still filling the role? Does the reasoning behind the classification still hold? The logs also show which AI services that are actually in use are missing from the inventory. A decision on these has to be made after the fact, and it is worth finding out why no request was submitted for them.

The results of the review go to management, and where experience calls for it, the policy and the decision rights are revised. The updated inventory and the record of decisions also document what the organization has done, should an internal or regulatory investigation ask.

Related service: Shadow AI assessment with an AI usage policy →

Contact

Ready to secure your critical systems?

Our experts will assess where your security stands and build a roadmap that fits your organization.

ARLITECH's office in Gödöllő

Visit us

You are welcome to visit our Gödöllő office in person.

Address
Centrum Office Building, 2100 Gödöllő, Dózsa György út 13, 3rd floor, office 309
Office hours
Mon–Fri, 9 a.m. to 5 p.m.
Get directions (opens in a new tab)