The New Rules for the World’s Biggest AI Models

[00:00:00] [music]
[00:00:04] Welcome back to another episode of Tech
[00:00:06] Unhinged, where tech gets human powered
[00:00:07] by code district. My name is Rabia Jawed
[00:00:10] and my guest today is Risto, who is the
[00:00:12] head of European policy and research at
[00:00:14] the future of life institute. For the
[00:00:16] past several years, Risto has been at
[00:00:18] the center of how the world governs
[00:00:20] advanced AI and has helped shape the
[00:00:22] general purpose AI and systematic risk
[00:00:24] provisions in the A EU AI act. He has
[00:00:27] also worked with the European
[00:00:28] Commission, the World Economic Forum and
[00:00:30] Stanford and runs one of the most read
[00:00:32] independent resources on the EUI act
[00:00:35] reaching over 50,000 subscribers. He’s
[00:00:37] also co-authoring forthcoming book, The
[00:00:39] AI Endgame. RTO, welcome to the show.
[00:00:42] >> Thanks so much for inviting me.
[00:00:43] >> You know, today we are talking about the
[00:00:45] new rules for the world’s biggest AI
[00:00:48] models. So, when we say the world’s
[00:00:50] biggest AI models, which ones do we
[00:00:52] actually mean and how many are there?
[00:00:55] Yeah, good question to start with. So in
[00:00:57] the EU act there is a definition of
[00:01:00] chapai. I think it’s article three which
[00:01:03] has a bunch of definitions. You can go
[00:01:05] through the list. It’s quite long. And
[00:01:07] uh essentially the definition is about
[00:01:09] general purpose models being models that
[00:01:11] can be used for a wide range of distinct
[00:01:14] tasks. Uh they uh use a lot of data and
[00:01:17] use self-s supervision at scale. And
[00:01:20] essentially that’s that’s the kind of in
[00:01:22] a nutshell uh what it is about. And
[00:01:24] there are different approaches to
[00:01:27] operationalizing those definitions. So
[00:01:29] if if I were to categorize AI, I would
[00:01:32] come up with two categories essentially.
[00:01:34] One would be standard general purpose
[00:01:36] models and then the other would be
[00:01:38] general purpose models with systemic
[00:01:40] risk. And these are the world’s largest
[00:01:41] models as you when you started talking
[00:01:43] as you put it. So for general purpose
[00:01:45] models with systemic risk, there’s a
[00:01:47] compute threshold. So how much compute
[00:01:50] is used to train these models that
[00:01:52] determines whether they count as purpose
[00:01:54] models with systemic risk or whether
[00:01:56] they are under that kind of a normal
[00:01:58] standard purpose models and that that
[00:02:00] threshold is 10 to the^ of 25 flop which
[00:02:03] my understanding is it’s still a very
[00:02:07] high threshold so very few models go
[00:02:10] over that and and so the European
[00:02:13] Commission itself they say that
[00:02:15] currently only a handful of companies
[00:02:17] develop these kinds of biggest most
[00:02:19] impactful jump models and future of life
[00:02:22] institute where I work they published
[00:02:24] this bianual report called the AI safety
[00:02:26] index and we just had one come out
[00:02:28] recently where we evaluate the safety
[00:02:32] practices of I think eight or nine
[00:02:34] companies some of the biggest from from
[00:02:36] different regions from China from the US
[00:02:38] and one also from Europe and I would say
[00:02:41] that maybe that list more or less
[00:02:43] captures this jumpi models with systemic
[00:02:46] risk some of these largest models. But
[00:02:48] you can also look at research conducted
[00:02:50] by for example epoch AI where they have
[00:02:54] put together lists of uh these according
[00:02:56] to these compute thresholds and they
[00:02:58] will also find you know handful of
[00:03:00] companies handful of models maybe dozens
[00:03:01] of those uh which will be captured by by
[00:03:04] this threshold and and all the other
[00:03:06] models they will there will be lots of
[00:03:08] other models that are so-called standard
[00:03:09] jump models but those ones under the U
[00:03:12] act will have fewer obligations they
[00:03:15] will have only limit obligation.
[00:03:17] compared to the biggest models which
[00:03:19] have more substantive obligations and so
[00:03:22] yeah we don’t know for sure the European
[00:03:24] Commission has not published a list yet
[00:03:25] but I would say based on the best
[00:03:27] knowledge we have a handful of models
[00:03:29] handful of companies and many of us know
[00:03:31] those companies they like we use their
[00:03:33] models their systems they’re always in
[00:03:35] the news and essentially those are the
[00:03:37] biggest AI models
[00:03:39] >> well yeah makes a lot of sense and you
[00:03:40] know we’ll just leave it to our
[00:03:42] listeners to figure that out for
[00:03:43] themselves
[00:03:44] >> well well we we can give some indication
[00:03:45] we can name some of those companies is I
[00:03:47] guess Antropic, Open AI, Google Deep
[00:03:49] Mind, XAI, Meta, Chinese Site, Deepseek,
[00:03:53] Alibaba, Zippo AI, you know, these are
[00:03:56] the types of actors.
[00:03:57] >> Makes a lot of sense. Resto, in 2019,
[00:03:59] you were a volunteer podcaster
[00:04:01] interviewing Jan Telen about whether AI
[00:04:05] could go badly wrong. Two years later,
[00:04:07] you joined the institute he co-ounded.
[00:04:09] How’s the hypothesis doing now? Yeah,
[00:04:12] back then I think in 2019 I think people
[00:04:16] were debating whether you know how
[00:04:19] capable AI was was going to be, when was
[00:04:22] it coming and yeah I mean I think a lot
[00:04:25] was about the capability how much how
[00:04:27] capable these AI model systems could be
[00:04:29] one day and we still have that debate to
[00:04:32] some extent but I think our debate is
[00:04:34] much more about for example how fast
[00:04:36] might some of the most serious dangerous
[00:04:39] capabilities arrive or when might
[00:04:42] certain kind of economic benefits emerge
[00:04:44] or come about there’s much more about
[00:04:46] how fast things are how controllable AI
[00:04:49] is how to govern it how can we get the
[00:04:51] benefits it’s not as much think about
[00:04:53] like you know skepticism about whether
[00:04:55] AI could be capable there is still some
[00:04:57] of that but I would say it’s it’s
[00:04:59] definitely less than in 2019 AI is much
[00:05:02] more real back then like we can go even
[00:05:04] a couple of years further not even to
[00:05:06] think about 2019 because that was way
[00:05:07] too early to some extent but for example
[00:05:09] back in 21 say when the EU act draft
[00:05:14] that was released by the European
[00:05:16] Commission back then we were just
[00:05:17] arguing with people about whether large
[00:05:20] language models were becoming a thing
[00:05:22] and people didn’t really expect that we
[00:05:24] were pointing towards for example GPD3
[00:05:26] by open AI were arguing that hey it
[00:05:29] looks like large language models could
[00:05:30] actually become a quite a big thing but
[00:05:31] lots of people were not persuaded they
[00:05:33] were like show me the evidence what are
[00:05:35] the numbers I actually ran through some
[00:05:37] numbers on crunch base I remember
[00:05:39] gathering some data seen that most of
[00:05:41] the investments looked like that they
[00:05:43] were going into self-driving vehicles
[00:05:44] rather than large language models for
[00:05:46] example. So yeah, a lot of discussion
[00:05:48] was around like how could the field even
[00:05:50] even progress and move. I think now
[00:05:52] things are different and I would say
[00:05:54] that compared to 2019 when we were
[00:05:56] talking about some of the risks, lots of
[00:05:57] people maybe would consider those risks
[00:06:00] highly speculative and as you started
[00:06:02] saying you use the word hypothesis.
[00:06:05] Somebody could use the word these are
[00:06:06] hypothetical scenarios or risks and
[00:06:09] things like that. I think some people
[00:06:11] are moving around that ter you know
[00:06:14] changing their mind updating because of
[00:06:16] some of the recent news that have been
[00:06:18] present in the news cycle quite a lot
[00:06:20] including the open AI incident with
[00:06:22] hugging face where essentially AI models
[00:06:25] I think it was GBD 5.6 six. Although
[00:06:28] let’s let’s caveat that they didn’t use
[00:06:30] their best safeguards uh for for those
[00:06:33] tests but they essentially got out of a
[00:06:36] sandbox environment hacked onto third
[00:06:38] party infrastructure which was hugging
[00:06:41] face in order to improve basically
[00:06:43] benchmarks and test scores and uh
[00:06:46] evaluations. So yeah some people are
[00:06:48] kind of changing their mind and seeing
[00:06:49] that maybe actually the some of these
[00:06:50] risks are not so hypothetical they’re
[00:06:52] not so speculated after all. Yeah, it
[00:06:54] took us you know when you mentioned 2019
[00:06:57] it’s uh it’s been 7 years but yeah I
[00:06:59] think that’s that’s how the landscape
[00:07:01] has changed quite a bit.
[00:07:02] >> Yeah. Yeah. No and we can quite
[00:07:03] evidently see that too. So you know to
[00:07:05] give a bit more context to the topic
[00:07:07] that we are discussing today Risto we
[00:07:09] want to dive a bit more into the why and
[00:07:11] not the what. So you know what made
[00:07:14] Europe decide AI needed its own set of
[00:07:16] laws.
[00:07:17] >> Yeah I think the the best answer to this
[00:07:20] question probably comes like straight
[00:07:22] from the act. If you look at it uh so
[00:07:24] there is under the EU act there’s
[00:07:26] information about the objective of this
[00:07:28] regulation like what’s the purpose why
[00:07:29] was this regulation even uh set up in
[00:07:31] the first place quite obvious as well
[00:07:33] that one the goal was to improve the
[00:07:36] functioning of the internal markets so
[00:07:38] the EU market EU market consists of 27
[00:07:41] countries it’s a large quite fragmented
[00:07:43] markets all the countries or most of the
[00:07:46] countries have their own languages their
[00:07:48] own cultures their own laws they they
[00:07:50] have quite a lot of differences uh
[00:07:52] between each other and and this law was
[00:07:54] proposed to improve that also improve
[00:07:57] the uptake of trustworthy AI. there was
[00:08:00] this hypothesis and and still is and I
[00:08:03] think it’s also not only a hypothesis I
[00:08:05] think there is you know some merit to it
[00:08:06] for sure that people don’t want to
[00:08:08] uptake people companies don’t want to
[00:08:10] uptake the technology unless they can
[00:08:12] trust it and in order to do so there has
[00:08:14] to be a high level of protection of
[00:08:16] health safety fundamental rights all
[00:08:19] these important values for the EU and if
[00:08:21] those values are protected properly
[00:08:24] through a regulation like this then
[00:08:26] people will want to use AI much more
[00:08:28] companies will want to use AI more. So I
[00:08:31] think that’s that’s that was really why
[00:08:33] this law was set up and and I would
[00:08:35] argue that considering other countries I
[00:08:37] think are converging towards setting up
[00:08:39] regulatory framework and and wanting to
[00:08:42] regulate AI but not necessarily always
[00:08:44] knowing how to do so. I I think it looks
[00:08:46] like the EU approach at least to some
[00:08:48] extent by being early there and starting
[00:08:51] to like exercising foresight and
[00:08:52] starting to work on it quite early
[00:08:54] actually makes sense because other other
[00:08:55] actors other regions they right now wish
[00:08:58] they also had some framework because
[00:09:00] they don’t always know how to deal with
[00:09:02] some of these incidents and whereas the
[00:09:04] EU might be ahead from that perspective.
[00:09:06] Yeah, you know that makes sense. But
[00:09:08] then you know there’s also a lot of talk
[00:09:10] around this Risto and you know we see
[00:09:12] that America built these models. So does
[00:09:14] you know China but it was Europe that
[00:09:17] wrote the rule book you know first and
[00:09:20] some people say that’s you know real
[00:09:22] leadership but others say Europe only
[00:09:24] regulates because it doesn’t have the
[00:09:26] big AI companies to compete with. So
[00:09:28] what is your thought process on that?
[00:09:30] >> I think it’s definitely leadership on
[00:09:32] the regulatory side. There is this
[00:09:35] concept that is often times used about
[00:09:38] the EU called the process effect.
[00:09:39] Whatever the regulation that the EU
[00:09:42] develops often times gets copied
[00:09:44] elsewhere in the world either by
[00:09:46] companies changing their practices in
[00:09:48] order to comply with the EU regulations
[00:09:50] in order to access the market goods and
[00:09:52] services for the market or other
[00:09:54] jurisdictions just copying the laws
[00:09:56] themselves uh and and getting
[00:09:58] inspiration from the laws. I think we we
[00:10:00] can definitely see some of that
[00:10:01] leadership in practice. For example,
[00:10:03] Korea recently introduced their AI law.
[00:10:06] It has some similarities with the EU
[00:10:09] act. And there are other places in the
[00:10:10] world that are definitely inspired by by
[00:10:13] the EU act. So definitely there’s some
[00:10:15] some leadership part. It’s important to
[00:10:17] to govern this technology and if those
[00:10:19] rules are copied elsewhere, then it also
[00:10:22] provides an advantage to the EU for the
[00:10:24] reason that we are really in a global
[00:10:25] market. We want to do business
[00:10:27] everywhere with other countries. And so
[00:10:29] it’s useful to set up some rules that if
[00:10:31] you have European companies really
[00:10:33] understanding those rules, they can and
[00:10:35] those rules are transferred elsewhere in
[00:10:38] other jurisdictions, then if you want to
[00:10:40] move to another place, you already
[00:10:42] understand those rules quite well and
[00:10:43] you can just comply with the same set of
[00:10:45] rules. You don’t have to reinvent
[00:10:46] something. So actually from that
[00:10:47] perspective, it’s useful to converge on
[00:10:50] similar types of rules everywhere. I
[00:10:52] will though say that regulation is not
[00:10:54] the only thing of course that matters
[00:10:57] when you talk about leadership you also
[00:10:59] want to talk about like general policy
[00:11:01] industrial policy you want to talk about
[00:11:03] building AI building different products
[00:11:06] and services so that’s a different part
[00:11:08] of it but I don’t think the AI act
[00:11:10] really gets in the way of that other
[00:11:12] part you just have to do industrial
[00:11:14] policy as well at the same time but you
[00:11:16] have to do AI regulation as well like
[00:11:18] both are important and I will also point
[00:11:20] to in the EU AI there are different
[00:11:22] measures and provisions that are there
[00:11:25] explicitly to support companies rather
[00:11:28] than somehow hinder them. So for example
[00:11:30] there are we’ve already talked a little
[00:11:32] bit about sandboxes. There are different
[00:11:34] sandboxes which um sandbox um concepts
[00:11:37] and ideas which basically are regulatory
[00:11:39] environments where companies could test
[00:11:41] out their products and get advice from
[00:11:43] regulators and kind of test things out
[00:11:44] and and play with things. So, so that’s
[00:11:46] that’s an example of a supportive
[00:11:48] measure and there are some other things
[00:11:49] there as well. But but yeah, I actually
[00:11:51] heard nice things a nice thing about
[00:11:54] from an MEP member of European
[00:11:56] Parliament Sergey Lagodinski who I think
[00:12:00] recently said that what remains missing
[00:12:03] for the EU is uh the capital to finance
[00:12:06] our own alternatives when it comes to to
[00:12:09] AI and yeah the AI act is like a
[00:12:12] different topic. If you want to talk,
[00:12:13] you can talk also about industrial
[00:12:15] policy and kind of what can be done to
[00:12:17] support that and there are initiatives
[00:12:18] in the EU on the that front as well.
[00:12:20] >> Yeah. No, makes a lot of sense. So, you
[00:12:22] know, rest of these rules have been on
[00:12:24] the books for a year now and on August
[00:12:26] 2nd the European Commission finally got
[00:12:28] the power to find the companies who are
[00:12:31] breaking them. Since then, has a single
[00:12:33] company been fined, investigated or even
[00:12:36] formally asked a question? So you know
[00:12:38] what we are trying to understand here is
[00:12:40] the implementation of these laws and you
[00:12:42] know how true is it to the core?
[00:12:44] >> Yeah I mean it’s a good question because
[00:12:45] it’s one thing to draft the law and have
[00:12:48] paperwork another thing to implement
[00:12:49] this. So it’s not enough to just draft
[00:12:51] it and then forget about it not
[00:12:53] implement it. I think I would be
[00:12:54] interested in following the whole
[00:12:55] process from from the conception of the
[00:12:58] the law from the drafting from the
[00:13:00] negotiation between different
[00:13:01] democracies the the process has taken a
[00:13:03] lot of time but but it’s not in by no
[00:13:05] means finished. So the AU act was passed
[00:13:08] in 2024. Now we’re in 2026. It’s already
[00:13:11] 2 years of certain implementation work
[00:13:13] and it’s still continuing. For example,
[00:13:14] the high-risk AI rules, they were
[00:13:16] delayed to 2027. So so this this work
[00:13:20] will continue to 27 to 28 and so on.
[00:13:23] That’s the first thing to say that I
[00:13:25] agree with with the premise that it’s
[00:13:27] important to to make sure that the law
[00:13:28] is actually implemented. But to your
[00:13:30] question concretely, first of all, I I
[00:13:33] work at a nonprofit, so I don’t really
[00:13:35] have access to all the information and
[00:13:38] the insight into how the European
[00:13:40] Commission operates, how the EU office
[00:13:43] operates, require public information. I
[00:13:45] require public scrutiny, journalists to
[00:13:47] scrutinize things, other actors to
[00:13:49] scrutinize what’s going on. And based on
[00:13:51] the information that I have publicly,
[00:13:52] there is no evidence of any kind of
[00:13:54] fine, any kind of request for
[00:13:56] information, any formal investigation
[00:13:58] into any companies, none of that. So the
[00:14:00] European Commission, the U office now
[00:14:02] has indeed these powers. So for one year
[00:14:05] already jump rules have actually been at
[00:14:08] play. But now from 2nd of August, there
[00:14:10] are these powers and and these powers
[00:14:12] indeed include fining companies. That’s
[00:14:14] that’s one option. But there are these
[00:14:16] other less serious options, so to speak.
[00:14:19] less escalatory options which are
[00:14:21] requesting information. What are you as
[00:14:23] a company doing to assess and mitigate
[00:14:25] systemic risks from your model? You can
[00:14:27] ask that uh more formally. From what I
[00:14:29] understand, for example, in in January
[00:14:32] this year, there was this um uh
[00:14:34] signature task force set up. There are
[00:14:36] companies companies who signed the code
[00:14:38] of practice. The code of practice is
[00:14:40] this voluntary tool that companies can
[00:14:42] use to comply with the EUA jump rules.
[00:14:45] And these these companies that signed
[00:14:47] the code of practice they have gotten
[00:14:49] together in the same room with
[00:14:50] regulators and have had dialogues with
[00:14:53] them from January already based on the
[00:14:56] information that that I have. So so that
[00:14:58] already has happened that that one we
[00:15:00] have information about but in terms of
[00:15:01] any concrete things that have been done
[00:15:04] we don’t have any information. So 2nd of
[00:15:06] August when these powers were were given
[00:15:08] to the AI office uh there was an
[00:15:10] announcement by the European Commission
[00:15:12] that these enforcement powers have
[00:15:14] kicked in and and these are things that
[00:15:17] are possible to do including for example
[00:15:20] um yeah the fines as we talked and so on
[00:15:23] also the European Commission has I think
[00:15:25] indicated that that they will use these
[00:15:27] powers kind of in a step-by-step
[00:15:29] fashion. So first they will start these
[00:15:32] dialogues and then they will ask uh
[00:15:34] request information more officially and
[00:15:36] if that doesn’t work through that
[00:15:38] information request they might ask
[00:15:39] certain kind of mitigation measures to
[00:15:41] be put in place or certain kind of
[00:15:42] remedies by the companies. If that
[00:15:44] doesn’t work then they might find
[00:15:46] companies but the finding part is is
[00:15:48] sort of like a last resort. They
[00:15:50] actually also have one other tool in
[00:15:51] their toolkit which is not allowing
[00:15:53] access to the market. So they they could
[00:15:55] ask uh the company not to just release
[00:15:57] the model on the market at all and they
[00:15:59] could also withdraw ask a model to be
[00:16:01] withdrawn from the market. So they they
[00:16:03] have these serious uh tools but yeah
[00:16:05] there’s no expectation that they will
[00:16:06] use this from the get- go. I do wish and
[00:16:09] I do want that they yeah officially
[00:16:11] request information from the companies
[00:16:13] and be in like formal conversation with
[00:16:15] them and also share some information
[00:16:17] about that publicly at least that other
[00:16:19] stakeholders are aware that they are
[00:16:21] enforcing the law and this law is not
[00:16:23] just piece of paper.
[00:16:24] >> So resto every company that is
[00:16:26] supposedly building with one of the big
[00:16:28] AI models is supposed to produce around
[00:16:30] like four documents. What are they and
[00:16:32] why do they matter? Yeah, there are
[00:16:34] these u yeah these different documents
[00:16:36] uh technical documentation downstream
[00:16:39] documentation. are giving information to
[00:16:41] downstream providers who are building on
[00:16:43] top of the CH models including you know
[00:16:46] understanding the capabilities of their
[00:16:48] models the limitations how to integrate
[00:16:50] it the requirements things like that
[00:16:52] there’s also copyright policy
[00:16:54] identifying and respecting rights
[00:16:57] reservations and also public training uh
[00:17:00] content summary so giving a summary of
[00:17:03] uh what kind of data you use to train
[00:17:05] your models those are the types of
[00:17:07] things that need to be done by all jumps
[00:17:09] model providers. There’s some nuances
[00:17:12] regarding uh when you’re open- source
[00:17:14] provider versus closed source. There are
[00:17:16] nuances around that. Why do this matter?
[00:17:18] I would say that some of this
[00:17:19] information could make the model more
[00:17:22] auditable, more understandable. For
[00:17:24] example, technical documentation,
[00:17:26] downstream documentation. It’s important
[00:17:28] for downstream providers to understand
[00:17:30] when they are using a jump model what
[00:17:32] can they do with the model what cannot
[00:17:34] they do what has the upstream done to
[00:17:37] make sure that the model is safe so so
[00:17:39] things like that also yeah rights
[00:17:41] holders interest considered properly
[00:17:43] that’s very important that’s that’s why
[00:17:45] the copyright policy is there I would
[00:17:47] say that though unfortunately I think
[00:17:49] among these four documents that you
[00:17:51] mentioned basically almost none of them
[00:17:53] I think are going to be public only the
[00:17:56] public training content summary is
[00:17:58] something that external stakeholders
[00:18:00] could could also scrutinize. So, so a
[00:18:03] lot of this is really dependent on the
[00:18:05] regulators themselves to make sure that
[00:18:07] they do great work, which is why it’s
[00:18:09] important AI offices set up with the
[00:18:11] right experts and the right capacity.
[00:18:14] Yeah.
[00:18:14] >> So, you know, um let’s talk a bit about
[00:18:16] the scope of this. If we take a company
[00:18:18] in San Francisco with no office in
[00:18:21] Europe, nobody on a payroll in the
[00:18:23] Europe, they put their model online
[00:18:25] where anyone can use it, download it,
[00:18:27] does European law reach them and if they
[00:18:29] do then how?
[00:18:30] >> So yeah, one difference one distinction
[00:18:32] here that I already made as well
[00:18:34] probably is is open source. So for
[00:18:37] example when we when we start talking
[00:18:39] about or when we talk about more of the
[00:18:41] systemic risk requirements which is
[00:18:42] which are the type of requirements that
[00:18:44] I care about which fall on the largest
[00:18:46] companies which is essentially the title
[00:18:48] of this conversation the biggest models
[00:18:51] in the world. uh in that case I I’m
[00:18:53] interested in yeah the distinction
[00:18:54] between open source and u and closed
[00:18:57] source and it’s interesting to to see
[00:18:59] that when when you’re a standard chopi
[00:19:02] model provider then indeed you have
[00:19:05] fewer requirements than u when you’re
[00:19:08] open- source provider versus when you’re
[00:19:09] a close source provider but when you’re
[00:19:11] doing developing these really really big
[00:19:13] models the most impactful models then
[00:19:15] you don’t really have any such
[00:19:17] exemptions so so if you’re an open
[00:19:19] source smaller provider you don’t get
[00:19:20] out from for example assessing and
[00:19:22] mitigating systemic risk. You have the
[00:19:24] requirements all the same regardless of
[00:19:25] whether you’re open source or close
[00:19:27] source that’s like something something
[00:19:28] quite interesting for me to um realize
[00:19:30] but but yeah your point about you you’re
[00:19:33] a San Francisco company are you in scope
[00:19:35] or not? Well, we would first have to
[00:19:36] think about well are you a provider of a
[00:19:39] jump model? Are you a provider of a jump
[00:19:41] model with systemic risk or the standard
[00:19:43] jump model? Let’s say we we go through
[00:19:46] that exercise in terms of thinking about
[00:19:47] the compute threshold for example that
[00:19:49] you meet and and we figure out what kind
[00:19:52] of model you have. Is it is it something
[00:19:54] that that can do wide range of distinct
[00:19:56] tasks and so on so forth. So you go
[00:19:58] through that exercise and and then then
[00:20:00] you really uh then really what matters
[00:20:02] is whether your chap model is um is
[00:20:06] placed on the EU market. Is it is it
[00:20:08] used in the EU market? That’s that’s
[00:20:10] what ultimately matters. Uh and in case
[00:20:12] you don’t like you don’t have a presence
[00:20:14] in the EU, my understanding is that as a
[00:20:16] non-EU provider, you have to appoint an
[00:20:19] authorized representative in the union
[00:20:21] before you can place the model in the
[00:20:23] market. So if you really want to do
[00:20:25] business in the EU then then yeah you
[00:20:28] have to follow the the chamber of rules
[00:20:30] which is the chapter five under the AI
[00:20:32] act. So so yeah that’s that’s something
[00:20:33] that is important for you to then fully
[00:20:35] understand.
[00:20:36] >> Yeah and you know Russo you’ve touched
[00:20:38] upon this but you know um just to put it
[00:20:41] out there does Europe’s rule book
[00:20:43] towards AI does it have enough
[00:20:45] flexibility to become everyone’s rule
[00:20:48] book or is it just you know compliant
[00:20:50] within the 27 European countries? Uh
[00:20:53] yeah that’s an interesting question. So
[00:20:55] I think when it comes to these general
[00:20:57] purpose rules when you look at what they
[00:21:00] actually demand and if you look at also
[00:21:02] the the code of practices that I I
[00:21:04] mentioned that have been developed to to
[00:21:06] help companies um comply with the U act
[00:21:08] rules those really were developed
[00:21:10] following industry best practices. They
[00:21:11] were not random rules developed out of
[00:21:14] nowhere. There were I think 13 experts
[00:21:16] uh that were set up by the EUI office
[00:21:19] and EUI office facilitate uh their work.
[00:21:22] But these independent experts from
[00:21:24] computer science from some social
[00:21:26] sciences they came together they
[00:21:28] reviewed what what does the literature
[00:21:30] right now say about how to govern AI
[00:21:32] including what do these companies
[00:21:33] themselves follow? We mentioned
[00:21:35] Antropic, OpenAI, uh these different
[00:21:38] companies. What are their risk
[00:21:39] management frameworks? What are their
[00:21:41] voluntary commitments? What kind of
[00:21:43] things are they doing? And just taking
[00:21:44] those and putting that together into
[00:21:46] this voluntary tool. That tool doesn’t
[00:21:47] have to be followed. It’s a voluntary
[00:21:49] thing. It’s it’s a tool for compliance.
[00:21:50] Uh these companies can use other kinds
[00:21:52] of measures instead if they want, but
[00:21:54] they have to abide abide by the uh the
[00:21:57] EU action rules. And these rules are
[00:22:00] like quite general. There are things
[00:22:02] like you have essentially you have to
[00:22:03] assess and mediate systemic risks. You
[00:22:05] have to ensure cyber security for your
[00:22:07] model. You have to conduct evaluations
[00:22:09] things like that. These are normal
[00:22:11] practices. These are becoming normal
[00:22:12] practices in the industry. And if there
[00:22:15] are companies that are not doing these
[00:22:16] things then the other actors and the
[00:22:18] best practices with this law and with
[00:22:20] the code of practice they kind of uplift
[00:22:22] uh the the quality of the field so that
[00:22:25] more actors would reach this standard.
[00:22:27] uh so but they they have not been
[00:22:29] invented from out of nowhere they have
[00:22:31] been developed following the the best
[00:22:33] practices from the field so I would say
[00:22:35] from that perspective it really isn’t
[00:22:37] unreasonable it’s quite very reasonable
[00:22:39] and if you look at different regions and
[00:22:42] whether they are already using some of
[00:22:44] these ideas from this rule book I would
[00:22:46] say they are and they’re converging to
[00:22:48] some extent for example you can think
[00:22:50] about California’s SB53 which is a new
[00:22:54] law and New York raise act and there’s
[00:22:56] in Illinois there’s a new safety law
[00:22:58] Illinois one for example also has
[00:23:00] pre-eployment model audits evaluation
[00:23:03] requirements so yeah you’re seeing
[00:23:05] convergence towards some of these ideas
[00:23:07] so there it’s not really oh just Europe
[00:23:09] following it other regions are also
[00:23:11] developing similar types of ideas
[00:23:12] >> so resto Europe decides which models are
[00:23:14] dangerous by how much computing power
[00:23:17] went into training them not by what they
[00:23:20] can actually do is that a good
[00:23:22] regulation or the only thing the room
[00:23:24] could agree on
[00:23:25] >> I think it’s part of a good regulation
[00:23:28] because we we do find that compute
[00:23:30] thresholds, how much cumulative compute
[00:23:33] was used, is part of what makes a model
[00:23:36] very capable, but it’s still a proxy.
[00:23:38] It’s it’s a good proxy in terms of being
[00:23:40] very measurable. You can verify Xanti.
[00:23:44] Um Xanti meaning before a model is
[00:23:46] trained, you already have an idea of if
[00:23:49] if a [clears throat] model is trained on
[00:23:50] this much compute, then it could be
[00:23:52] capable of these and these uh things.
[00:23:54] Hard to game. It’s a very yeah it’s a
[00:23:56] quantifiable thing and as I mentioned
[00:23:59] you can you can decide it before
[00:24:00] deployment capability threshold which
[00:24:02] would track the actual thing that we
[00:24:03] care about or some kind of risk
[00:24:05] measurements so solid and so clear and
[00:24:08] and the field is still working how to
[00:24:09] figure those things out how to measure
[00:24:11] capabilities how to evaluate uh concrete
[00:24:13] risks those are more difficult to figure
[00:24:16] out I would also as a last thing I would
[00:24:17] say is the EU act is not just about
[00:24:19] compute thresholds this this can be a
[00:24:22] misunderstanding if we think that it’s
[00:24:24] only about comput thresholds. There’s
[00:24:25] actually article 51 which talks about I
[00:24:28] think the classification of models into
[00:24:31] systemic risk category. Basically the
[00:24:33] idea there is that yes you can use
[00:24:35] compute threshold I mentioned the 10 to
[00:24:36] the^ of 25 flop but also if we have
[00:24:40] other evaluations of its high impact
[00:24:42] capability of of a jump model that could
[00:24:45] be used also the commission can can
[00:24:48] designate a model systemically risky for
[00:24:50] example through a qualified alert by an
[00:24:53] institution called the scientific panel
[00:24:56] there is this institution set up I think
[00:24:58] it’s 60 experts in a group and and these
[00:25:01] experts can alert the AI office, hey,
[00:25:03] have you considered that like this model
[00:25:05] could be systemically risky? We have
[00:25:07] this and this evidence and these reasons
[00:25:09] that could be used. Also, there is um in
[00:25:11] the appendix there is also um some extra
[00:25:14] criteria that can be used to determine
[00:25:16] whether a model is systemic risky
[00:25:18] including how many users it has and
[00:25:20] things like that. So, so really impact
[00:25:22] can potentially be determining quite a
[00:25:25] lot of criteria not just compute even
[00:25:27] though compute is the the most like
[00:25:29] concrete proxy. some time ago when the
[00:25:31] industry complained these rules were not
[00:25:33] ready, you implied it as a propaganda by
[00:25:36] them, you know, while I was going
[00:25:38] through your LinkedIn. So since then,
[00:25:39] half the rules got delayed until
[00:25:41] December 2027 and Meta refused to sign
[00:25:44] the code of practice and paid nothing
[00:25:46] for it. So a CTO who’s listening to this
[00:25:48] conversation right now, what are they to
[00:25:51] conclude from this? One thing that I
[00:25:53] have written about for example in tech
[00:25:55] policy press I wrote the piece with
[00:25:58] Laura Coroli together she was very close
[00:26:02] to the drafting of the act she worked
[00:26:04] for um MEP Brando Benfe when benef was a
[00:26:08] co-papertor of the act was from the
[00:26:11] European Parliament uh you know critical
[00:26:13] stakeholder behind drafting the act and
[00:26:15] later Laura Cley became independent
[00:26:18] researcher and moved out from the
[00:26:20] European Parliament but followed I act
[00:26:22] very closely still and um we during when
[00:26:25] the code of practice was developed for
[00:26:27] JavaSci, we noticed there was um I I
[00:26:30] would say yeah propaganda by some
[00:26:32] industry actors. They were claiming
[00:26:34] actually only a few days before the code
[00:26:37] of practice was finalized and published.
[00:26:39] They said that the code was still
[00:26:42] incomplete and still being debated and
[00:26:44] they asked for a 2-year pause on the EU
[00:26:48] Act. That was literally a few days
[00:26:49] before the code of practice was going to
[00:26:51] be finalized and there was public
[00:26:53] information that the code of practice
[00:26:54] was going to be released and it’s very
[00:26:56] very likely that they actually had this
[00:26:58] information. for for some reason they
[00:26:59] were spreading this m misinformation and
[00:27:01] so so that was something that we called
[00:27:04] out and and were annoyed by and it’s
[00:27:06] important to now in hindsight realize
[00:27:09] that yeah the code of practice was
[00:27:10] developed there were lots of guidelines
[00:27:12] for being drafted there have been
[00:27:16] institutions set up like the scientific
[00:27:17] panel that I mentioned there’s the
[00:27:19] signaturary task force where companies
[00:27:21] come together and they uh can discuss
[00:27:24] with regulators so a lot of things have
[00:27:26] actually been implemented around Jupai
[00:27:27] and the file has moved much much further
[00:27:30] and Jupai was never delayed. Uh there
[00:27:33] was a delay of high-risk rules uh that I
[00:27:35] mentioned earlier and one reason for
[00:27:37] that maybe the main reason for that is
[00:27:39] that technical standards are not ready.
[00:27:41] These are essentially similar to the
[00:27:43] code of practice compliance tools for
[00:27:45] companies to use to comply with the
[00:27:47] regulation. Um and yeah the technical
[00:27:49] standards for high-risisk AI rules are
[00:27:50] not ready. Hence there was a need to
[00:27:52] delay and and work more on these
[00:27:54] technical standards to make sure that
[00:27:55] companies can actually comply more
[00:27:56] easily with the law. It’s important to
[00:27:59] understand that a lot of progress has
[00:28:00] actually been made about general purp.
[00:28:02] And so your point regarding meta is
[00:28:05] correct that they haven’t been fined
[00:28:08] despite not signing the code of
[00:28:10] practice. There’s no penalty to refusing
[00:28:12] to sign the code of practice. The
[00:28:14] signing was voluntary. it doesn’t do
[00:28:15] anything but the duties they duties to
[00:28:18] comply with the general sales those
[00:28:20] apply regardless of what kind of method
[00:28:22] you use to comply it’s really probably
[00:28:25] in the best interest of a company to use
[00:28:26] the code of practice because the code of
[00:28:28] practice as I said has been developed
[00:28:30] with the perspective that we have all
[00:28:31] these best practices by companies let’s
[00:28:33] look at them and let’s bring them
[00:28:34] together whatever other methods
[00:28:36] alternative means meta or another
[00:28:38] company might use my guess would be that
[00:28:40] those means would be compared to the
[00:28:42] code of practice as well because that’s
[00:28:44] that’s what the field indicates I would
[00:28:46] think that meta would have to follow the
[00:28:47] code of practice to some extent at least
[00:28:49] as well to comply with the rules and
[00:28:51] maybe last thing regarding meta point uh
[00:28:54] European commission has in their
[00:28:56] guidelines said that may scrutinize
[00:28:58] companies more that have not signed the
[00:28:59] code of practice so that’s another
[00:29:01] reason why it actually might make sense
[00:29:03] to to be a signature and and to follow
[00:29:06] it and and regarding your your also your
[00:29:08] question around a CTO what should a CTO
[00:29:10] conclude well a CTO should really
[00:29:13] understand that in case the CTO is
[00:29:16] working in a company that is could be
[00:29:18] considered a downstream provider
[00:29:19] somebody who uses a general purpose
[00:29:22] model builds up on top of somebody
[00:29:24] else’s model say a general model with
[00:29:26] systemic risk on tropics models or open
[00:29:28] airs model it’s important to understand
[00:29:30] that from 2nd of August last year
[00:29:32] already these companies have had these
[00:29:34] rules that they have had to comply with
[00:29:36] and now from 2nd of August this year
[00:29:38] there are enforcement powers they could
[00:29:40] be potentially fined or information
[00:29:42] requested or so It’s important to
[00:29:44] understand that and it’s important to
[00:29:46] understand that when you have an AI
[00:29:47] system that you use to build on top of
[00:29:49] those models in case you have your own
[00:29:51] requirements under AI act or you might
[00:29:53] have in the future it’s important for
[00:29:55] you to understand what that model
[00:29:56] provider is doing to make sure that they
[00:29:58] comply with the regulations so that you
[00:30:00] won’t have any issues that you know
[00:30:01] sudden sudden issues that you didn’t
[00:30:03] really consider.
[00:30:04] >> So Russo we did talk about the bright
[00:30:06] side towards these laws but you know
[00:30:08] European developers they get frontier
[00:30:10] models pretty late or not at all. Claw 2
[00:30:13] never really arrived. Google’s chatbot
[00:30:15] launched 2 months behind and OpenAI’s
[00:30:17] first agent landed 7 weeks after America
[00:30:20] got it. So is that a price of these
[00:30:22] rules?
[00:30:22] >> I would um point people to do some
[00:30:24] research that has been conducted on this
[00:30:26] topic. I think there was um this think
[00:30:28] tank called CAVI and they recently
[00:30:31] published a research where they found
[00:30:33] that the best available data indicates
[00:30:35] that uh the delays of these so-called
[00:30:38] frontier models or top AI models they
[00:30:41] have been perhaps due to GDPR which is
[00:30:43] the data privacy regulation rather than
[00:30:45] the AI act. In fact they said that they
[00:30:48] don’t find any strong evidence that the
[00:30:49] EU act has caused delays or
[00:30:52] non-releases. But it’s also important to
[00:30:54] say that the AI act is you know has has
[00:30:56] not been started being implemented very
[00:30:58] much yet though when they did this
[00:31:00] research the the jump rules from 2nd of
[00:31:03] August 2025 already applied. So, so it
[00:31:06] was already there. That’s something to
[00:31:07] say. Also, I read that Gavaii they found
[00:31:11] uh they looked at releases from Meta,
[00:31:13] Google, Open Airropic and and found that
[00:31:16] in the case of the EU, there were 11%
[00:31:20] models were delayed or not released and
[00:31:22] in the case of the UK 7%. And yeah, as I
[00:31:24] mentioned, data protection was was one
[00:31:26] of the things that was mentioned there,
[00:31:27] but the main outlier there really was
[00:31:30] meta. Meta was the main one that was not
[00:31:32] releasing uh on the market. uh these uh
[00:31:35] these other companies they they really
[00:31:37] had a few issues like that and also
[00:31:39] there is a trend towards improving the
[00:31:42] so delays have fallen over time and when
[00:31:45] when I think about for example the very
[00:31:47] early phases of say Google’s bar models
[00:31:50] or think about entropic early claw and
[00:31:53] things like that then it was actually
[00:31:54] quite common and a lot of discussion
[00:31:56] around these delays releases release
[00:31:59] delays and a lot of debate over that
[00:32:01] recently there has been less now There
[00:32:04] is has also been a bit more with the
[00:32:06] recent these US export controls and
[00:32:10] things like that but these have been not
[00:32:11] due to EU’s regulation or anything like
[00:32:14] that that those have been due to
[00:32:15] national security concerns in the US.
[00:32:17] Yeah, it’s important to assess these
[00:32:19] delays and these access questions, but I
[00:32:22] would not also overclaim them based on
[00:32:25] the evidence that we have. And partially
[00:32:27] some of those delays indeed could be uh
[00:32:30] due to other regions, jurisdictions
[00:32:33] trying to figure out how to make sure
[00:32:34] that these models are safe for national
[00:32:36] security concerns and other concerns and
[00:32:39] putting uh frameworks in place, putting
[00:32:41] their own regulations in place to like
[00:32:43] properly deal with that. And uh but
[00:32:45] yeah, it’s important for Europe to
[00:32:47] consider if they do want to have access
[00:32:49] to these models. They might want to look
[00:32:52] at some of the aspects related to their
[00:32:54] data privacy regulation that might be
[00:32:56] useful to assess. But more importantly,
[00:32:59] I think it’s important to work on
[00:33:01] diplomacy, work on geopolitic, work on
[00:33:03] collaborating with other powers in the
[00:33:06] world and and make sure that there is
[00:33:08] leverage uh whether it’s regulatory
[00:33:10] leverage but also other kinds of supply
[00:33:12] chain leverage and other kind of
[00:33:14] leverage and partnerships so that these
[00:33:16] delays don’t occur uh in an unwanted
[00:33:19] way.
[00:33:20] >> So in July, OpenAI disclosed that one of
[00:33:21] its agents got out of its test
[00:33:23] environment and into hugging faces
[00:33:25] infrastructure. the safeguards had been
[00:33:27] switched off deliberately for the test.
[00:33:29] Is that a serious incident someone has
[00:33:30] to report or does the word test make it
[00:33:33] disappear?
[00:33:33] >> I don’t think the the EU’s regulation
[00:33:36] has some kind of exemption when when you
[00:33:39] have a test or you decide that you don’t
[00:33:41] want to put certain kind of safeguards
[00:33:42] on the model because you’re just testing
[00:33:44] it in a sandbox that doesn’t make
[00:33:46] obligations uh disappear. So there is uh
[00:33:50] article 55 under the UA act which
[00:33:52] requires the providers of systemic risk
[00:33:55] jump models to uh track documented
[00:33:58] report. There is uh incidents to the UI
[00:34:00] office and there is even a dedicated
[00:34:02] channel set up for it. It’s called the
[00:34:04] EU send and my understanding actually is
[00:34:06] that open AAI use uh that channel and
[00:34:10] actually reported it to the UF is the I
[00:34:13] think political for example political
[00:34:15] magazine political they have covered it
[00:34:17] and some other magazines as well. So so
[00:34:20] they they probably actually uh followed
[00:34:22] it to some extent. I I would also say
[00:34:23] that it’s um yeah at least my vision my
[00:34:27] my hope for the the law is and for the
[00:34:30] regulators is that they and for the
[00:34:31] companies as well that the companies
[00:34:32] don’t just medi try to put in
[00:34:34] mitigations after the fact or or correct
[00:34:37] things after harm has occurred. I would
[00:34:38] hope that they actually take some
[00:34:40] preventative action or if if companies
[00:34:42] don’t do that then then the the laws the
[00:34:45] uh the policy makers they require these
[00:34:47] preventative actions to be taken and in
[00:34:50] the case of this particular incident
[00:34:51] that you mentioned I think it’s
[00:34:52] important to understand that under the
[00:34:54] EU act uh the concept systemic risk
[00:34:57] applies and in the code of practice
[00:34:59] systemic risk is defined at least
[00:35:01] according to four specific risk
[00:35:03] categories these are cyber offense loss
[00:35:05] of control harmful manipulation and
[00:35:08] chemical biological risks. And many
[00:35:10] experts and commentators have analyzed
[00:35:13] these so-called rogue AI incidents
[00:35:16] recently during the open AI hugging
[00:35:17] phase example. Those are examples of
[00:35:19] both cyber offense and loss of control
[00:35:22] incidents and more recently or a bit
[00:35:25] less recently entropics myths was also
[00:35:27] in the news for cyber capabilities and
[00:35:29] and yeah the the UA has has these
[00:35:32] relevant provisions and the code of
[00:35:33] practice has these relevant uh tools to
[00:35:35] deal with that. So your own organization
[00:35:38] graded Mistral the worst AI company in
[00:35:41] the world this summer which is below
[00:35:43] Deep Seek below X AI. So you know does
[00:35:47] that mean the rules were really
[00:35:48] necessary to um you know uh apply or was
[00:35:53] was that Europe being over policying? I
[00:35:56] >> I think it’s an example of uh these
[00:35:58] safety issues really not being only
[00:36:01] relevant for like one region or
[00:36:03] something. let’s say only China safety
[00:36:05] issues are relevant only for China or
[00:36:07] only for the US. No, actually turns out
[00:36:10] based on this FLI AI safety index that
[00:36:13] out of the nine companies that we
[00:36:14] assessed, there were companies that did
[00:36:16] poorly in every uh region, right? In
[00:36:19] Europe, in the US, in China and so this
[00:36:22] is this was not just like one region’s
[00:36:24] issue and you mentioned some of these
[00:36:26] companies that that did poorly in the
[00:36:28] grades, but I would also say that none
[00:36:30] of these companies got A’s or B’s
[00:36:33] overall. So the distance between for
[00:36:35] example getting a C++ which entropic cut
[00:36:38] and F is not necessarily like a distance
[00:36:41] between having a safe model and having
[00:36:43] safe model. It’s it’s more of a degree.
[00:36:46] It means that for example, Mistral has
[00:36:48] not published a safety framework which
[00:36:50] we recommend that they do and they they
[00:36:52] don’t have publicly shared that they
[00:36:55] have a governance structure or they also
[00:36:57] don’t engage substantially with
[00:36:59] existential safety concerns and then
[00:37:01] their leadership actually is often times
[00:37:03] consistently downplays those concerns.
[00:37:05] They don’t do so well on safety
[00:37:08] benchmarks. That’s an improvement they
[00:37:10] should make. these recommendations we
[00:37:11] make to them and we make to some other
[00:37:13] companies as well but it’s not like
[00:37:15] entropic or open AI or Google deep minds
[00:37:17] models are completely safe uh I wouldn’t
[00:37:19] we definitely don’t make that case it’s
[00:37:21] also important to perhaps mention here
[00:37:24] that mistrial now recently has endorsed
[00:37:27] the code of practice has signed it the
[00:37:30] code of practice and it CEO has publicly
[00:37:33] also said that uh regulation was not
[00:37:35] Europe main problems uh main problem and
[00:37:38] the the rules I think the rules are
[00:37:40] necessary and and and it indicate that
[00:37:43] he doesn’t think that that’s the main
[00:37:45] problem of Europe either. But
[00:37:46] considering that Mist has signed the
[00:37:48] code of practice, we would also expect
[00:37:49] them to do better regarding what the
[00:37:51] code of practice requires. So yeah,
[00:37:52] that’s essentially what the FI safety
[00:37:54] index indicates. But uh this is by no
[00:37:57] means yeah endorsement of other
[00:37:58] companies safety practices. They’re not
[00:38:00] they have lots of room to improve as
[00:38:02] well.
[00:38:03] >> That was very thorough and thank you
[00:38:04] Risto for that. So you know we are
[00:38:07] moving towards some of the last
[00:38:08] questions for our conversation today.
[00:38:10] The office that enforces these rules
[00:38:12] across all of Europe has only about 145
[00:38:15] people the AI office and just a few
[00:38:18] dozen actually you know are deployed in
[00:38:20] evaluating the biggest AI models and
[00:38:23] since the fines they can hand out run
[00:38:25] into the billions. Can a team that small
[00:38:27] really take on the largest tech
[00:38:29] companies in the world and really make
[00:38:31] it stick to them?
[00:38:32] >> Uh the the jury is still out there. We
[00:38:33] haven’t seen any specific formal action
[00:38:35] being taken, right? I I’ve spoken about
[00:38:38] some of these dialogues and the
[00:38:41] enforcement powers and the provisions
[00:38:42] and so on, but there’s no formal action.
[00:38:44] I haven’t seen any requests for
[00:38:46] information being announced. I haven’t
[00:38:47] seen anything like that. So to be seen,
[00:38:49] we have to see. I think clearly the AI
[00:38:52] office needs to be their capacity needs
[00:38:54] to be improved and increased. They need
[00:38:56] more staff. Uh they have a lot of work.
[00:38:58] Some of these things that companies have
[00:39:00] to provide to them like the system cars,
[00:39:02] model reports, they’re massive pieces of
[00:39:04] information. There’s so much information
[00:39:06] there that needs to be analyzed against
[00:39:08] code of practice, the EU act rules.
[00:39:10] There’s a lot of work to be done. Uh so
[00:39:12] they need resources. Also, I think
[00:39:14] recently there were several members of
[00:39:16] the European Parliament, Ben Lagodinski,
[00:39:19] one Sparak Boss, many of them called for
[00:39:23] more resources to their office. There
[00:39:25] have been civil society organizations
[00:39:27] that have called for more resources.
[00:39:28] Yoshua Benjio who who was one of the
[00:39:31] experts behind the code of practice
[00:39:33] drafting and he has called publicly for
[00:39:35] more resources for the AI office. Very
[00:39:37] clear. I completely agree. I will say on
[00:39:39] the more positive side that the UR
[00:39:41] office just announced that they will
[00:39:43] hire 40 additional experts to mainly to
[00:39:46] the AI safety and the regulatory and
[00:39:49] compliance units to deal with the act
[00:39:51] enforcement issues uh that we have been
[00:39:53] speaking about. So that’s been positive.
[00:39:55] The deadline to apply is 8th of
[00:39:57] September. Any listeners that would be
[00:39:59] interested in working at the UA office.
[00:40:01] I don’t work at the UA office. I’m not
[00:40:03] able to provide information about uh how
[00:40:05] good of a place it is. But I can say
[00:40:08] that does very important work. It’s very
[00:40:10] important that they succeed with the
[00:40:11] enforcement of channels rules with
[00:40:13] systemic risk to avoid more of these
[00:40:15] kinds of incidents or or more of these
[00:40:17] incidents that we have talked about the
[00:40:18] open air hugging face incident as an
[00:40:20] example. I will also say that in
[00:40:21] addition to the extra resourcing or the
[00:40:24] new staff that’s going to be hired, they
[00:40:26] also recently announced that they have
[00:40:28] hired Alessandro Abat from Oxford
[00:40:31] University as their lead scientific
[00:40:34] adviser to support his scientific
[00:40:36] expertise on model evaluation and other
[00:40:39] kinds of things. So, so that’s that’s a
[00:40:40] welcome thing because that was delayed
[00:40:42] for for a while and now has been
[00:40:44] announced. And also I mentioned the EU
[00:40:46] scientific panel that has been set up to
[00:40:48] also support the AI act enforcement.
[00:40:49] That’s also positive. Yeah, overall they
[00:40:51] should definitely hire more people. As
[00:40:53] the last things here I will mention,
[00:40:55] they also have announced several
[00:40:56] important other resources relevant for
[00:40:59] general purpai including uh a
[00:41:01] whistleblower uh tool people uh from
[00:41:04] from these companies can in a secure
[00:41:06] manner blow the whistle to the UI office
[00:41:09] uh about practices uh in companies and
[00:41:12] also there’s a dedicated complaint uh
[00:41:14] channel for downstream providers. So we
[00:41:18] have talked about how there are these
[00:41:20] companies that might use a jump model
[00:41:22] and build uh something on top of it.
[00:41:24] They can complain to the EU office if
[00:41:26] they don’t think that the model that
[00:41:28] they are using the jump provider has
[00:41:30] like done enough safety practices. So
[00:41:32] there there’s dedicated channel to that
[00:41:34] as well. So I’m hoping yeah more things
[00:41:36] like this can support the the
[00:41:37] enforcement as well.
[00:41:38] >> Absolutely. And you know looking forward
[00:41:40] for them to um come out that way. So
[00:41:42] Resto your book comes out next year and
[00:41:44] it’s called the AI end game. Were you
[00:41:47] super like impressed by the Marvel
[00:41:50] series to give it that name? So maybe
[00:41:52] why this name if you can talk about it
[00:41:54] and if you can walk us through what this
[00:41:56] book is about without probably spoiling
[00:41:58] anything and what are you co-authoring
[00:42:01] in this particular book about and why
[00:42:02] should you know our listeners be excited
[00:42:04] about it?
[00:42:05] >> Definitely excited about all these
[00:42:06] Marvel movies. I just went to see the
[00:42:09] recent Spider-Man. So I’m definitely
[00:42:10] definitely a fan of those movies and and
[00:42:12] they definitely inspire me to some
[00:42:14] extent. But um I I think what I was and
[00:42:17] what we with my co-author were mostly
[00:42:19] most motivated by was there are a lot of
[00:42:21] books that talk about problems a lot.
[00:42:23] You can read like for example shout out
[00:42:25] to a book that recently came out uh
[00:42:28] titled if anyone builds it everyone
[00:42:30] dies. Very concrete title right like
[00:42:32] very very negative very pessimistic
[00:42:34] relatively short on solutions and thin
[00:42:36] on solutions. We are we want this book
[00:42:39] to be more optimistic. So we are
[00:42:40] thinking about okay we have all these
[00:42:42] pressing risks we have all these
[00:42:43] challenges. we have all these incidents
[00:42:45] that we’re seeing in the future we could
[00:42:46] have even more incidents because the
[00:42:48] capabilities are increasing it’s more
[00:42:50] widely deployed and so on so forth. So
[00:42:51] we thought, okay, what does an a proper
[00:42:54] endgame actually look like? Meaning
[00:42:56] basically a theory of victory that we
[00:42:57] could have a longerterm stable solution
[00:43:00] and we could get upsides from AI. We
[00:43:02] could we could mitigate the downsides.
[00:43:04] We could reduce these risks, some of
[00:43:06] these largest risks. We could at the
[00:43:07] very least delay them for a longer time
[00:43:09] so we could get the benefit. That was
[00:43:10] essentially what we wanted to go for.
[00:43:12] And this book will have lots of stuff
[00:43:14] about risks. So we start by first
[00:43:16] talking about the risks. So we we lay
[00:43:17] out different risk scenarios including
[00:43:19] describing those scenarios in like in
[00:43:21] concrete ways how like a particular
[00:43:23] scenario might might manifest but also
[00:43:25] more of the theory behind the risks and
[00:43:27] the evidence but then then we actually
[00:43:30] most of it we are relative positive
[00:43:32] about so or like positively minded about
[00:43:34] so so we have we want to give uh the
[00:43:37] readers some kind of tools concepts to
[00:43:39] think about these pressing risk and we
[00:43:41] also talk obviously a lot about endg
[00:43:43] games specific mitigations and specific
[00:43:45] ways to solve the issue And then we also
[00:43:47] uh finish the book with uh positive
[00:43:50] applications, beneficial AI, positive
[00:43:52] futures. We’re looking like we want to
[00:43:53] make it more positive. Marvel movies
[00:43:55] usually end with a with a positive.
[00:43:57] >> So Resto being on the side of you know
[00:43:59] the policy and being the policy makers.
[00:44:01] How do you guys stay closely in touch
[00:44:03] with the reality you know of the people
[00:44:05] who are inventing all of these biggest
[00:44:08] AI models so that you know whatever laws
[00:44:10] might come out of that they are
[00:44:13] applicable or they can be implemented at
[00:44:15] large. I mean who bridges that gap
[00:44:18] between policy and people the actual
[00:44:21] innovation.
[00:44:21] >> I would say that the the EU act was
[00:44:25] drafted on this principle and generally
[00:44:27] the EU works quite well from that
[00:44:28] perspective. So for example the EU act
[00:44:31] the draft was released in 2021 April and
[00:44:34] it took more than 3 years to pass the
[00:44:36] law. There was like a lot of negotiation
[00:44:38] back and forth. uh the EU apparatus for
[00:44:40] for the listeners who don’t know it it
[00:44:42] consists of basically three institutions
[00:44:44] that all debate with each other. So the
[00:44:46] European Commission releases a draft but
[00:44:48] then the European Parliament and the
[00:44:49] Council of the EU policy makers come
[00:44:51] together to amend the law and then the
[00:44:53] European Commission comes sits together
[00:44:54] with those other institutions and
[00:44:56] debates and and kind of improves upon
[00:44:59] the initial law and they bring together
[00:45:00] a lot of experts and interest groups. So
[00:45:03] so they speak with civil society, speak
[00:45:05] with industry, they speak with academia.
[00:45:07] So a lot of different voices are are
[00:45:09] heard and brought into this process.
[00:45:11] It’s by no means a perfect process. Of
[00:45:13] course there’s all kinds of issues
[00:45:15] regarding well maybe did the industry
[00:45:17] get too much or did like a certain civil
[00:45:19] society actor get too much air time or
[00:45:21] like was a particular idea given too
[00:45:23] much emphasis compared to some other
[00:45:25] ideas that always these kinds of
[00:45:26] concerns could be relevant to raise not
[00:45:28] perfect for example I mentioned the code
[00:45:30] of practice. The code of practice for
[00:45:31] jumpi was drafted leading led by 13
[00:45:36] experts and around my understanding is
[00:45:38] around thousand stakeholders to some
[00:45:40] extent were included in the process.
[00:45:42] Basically, you could apply to be part of
[00:45:43] the process and my understanding is like
[00:45:46] almost everybody was accepted that had
[00:45:48] some kind of AI governance related
[00:45:49] background and they could meet at a
[00:45:51] regular interval and provide input to
[00:45:53] that. I think several drafts like maybe
[00:45:55] three drafts or so were presented to
[00:45:57] those stakeholders and then they could
[00:45:59] provide feedback and then they could go
[00:46:00] back and like reddraft and change some
[00:46:02] things. You can imagine a scenario and
[00:46:04] usually that’s the case that you would
[00:46:05] have open air entropic and deep sea
[00:46:08] coming together and behind closed doors
[00:46:10] drafting their own you know toolkit and
[00:46:12] then they would follow that and they
[00:46:14] would say oh we are complying with the
[00:46:15] EU act. That was not what happened. It
[00:46:17] was much more democratic. I have all
[00:46:18] kinds of you know concerns and issues
[00:46:20] and and qules but but generally it’s a
[00:46:22] pretty nice democratic consensus uh
[00:46:25] focused process.
[00:46:26] >> So being a researcher yourself what are
[00:46:28] your go-to AI tools? my goto AI tools. I
[00:46:31] listen to a lot of podcasts. So I would
[00:46:33] recommend people to to pick up podcasts
[00:46:35] like for example Future of Life
[00:46:37] Institute if I may say myself has a
[00:46:39] really great podcast. Yeah, lots of lots
[00:46:41] of AI podcasts that I recommend. I mean
[00:46:43] I’m usually just reading a lot of
[00:46:45] academic papers. Uh so for example for
[00:46:47] the book and others I have set up Google
[00:46:49] Scholar alerts for like AI safety terms
[00:46:52] and things like that. So I’m getting
[00:46:53] lots of emails. I would also recommend
[00:46:55] people if you want to understand the U
[00:46:58] act better definitely check out our own
[00:47:00] tool we have the act explorer and
[00:47:02] website and newsletter lots of stuff
[00:47:03] that you can use so and I personally go
[00:47:06] to those every day because like some of
[00:47:07] these things that we talk about uh we
[00:47:09] got into like some higher level points
[00:47:11] but there way more details that you can
[00:47:13] get into and if you want like a proper
[00:47:15] legal interpretation legal assessment
[00:47:17] which my you know thoughts today were
[00:47:19] not then of of course you have to go and
[00:47:21] actually check the legal text and so on.
[00:47:23] So, so our tools can can support on that
[00:47:25] front as well. I’m also active on
[00:47:27] LinkedIn and Twitter. So, there’s lots
[00:47:29] of AI safety researchers there. Lots of
[00:47:31] policy makers overloaded with
[00:47:33] information to be honest. For example, I
[00:47:34] use a tool called Super Whisper.
[00:47:37] Basically, I could go for a walk and I
[00:47:39] could record my thoughts and get a
[00:47:41] transcript and it could stay on on my
[00:47:43] device. So, it would not go into any
[00:47:45] cloud or anything. So, more like privacy
[00:47:47] oriented uh applications. I use that
[00:47:49] tool a lot. So restra 7 years of arguing
[00:47:52] for these rules and you know being a
[00:47:54] part of making them what’s the hardest
[00:47:56] lesson that you had to unlearn? I don’t
[00:47:59] know if this resonates with many of the
[00:48:01] readers but maybe resonates with some or
[00:48:03] listeners with some listeners. I
[00:48:05] expected kind of policy work and policy
[00:48:08] research and policym to be more stable
[00:48:11] and more like linear for some reason
[00:48:13] like it was like a false uh false
[00:48:15] learning for me a false assumption but
[00:48:17] for example the EU act whole process was
[00:48:20] like very messy and very uh nonlinear in
[00:48:23] the sense that there are a lot of actors
[00:48:24] that are all involved and there might be
[00:48:26] a particular event or incident that
[00:48:29] might bring certain concerns into the
[00:48:31] picture or maybe there’s like some
[00:48:32] political consideration or Maybe the
[00:48:35] field of academic research learns
[00:48:37] something new or something goes out of
[00:48:39] fashion or something comes into fashion
[00:48:41] and all of those things could have an
[00:48:43] impact on policym. So you might be say
[00:48:45] you might be drafting a specific type of
[00:48:48] proposal or making a specific
[00:48:49] recommendation and then turns out in
[00:48:52] half a year it’s like completely blown
[00:48:54] apart. Nobody like follows that anymore.
[00:48:56] That that opportunity doesn’t exist
[00:48:58] anymore and there other opportunities.
[00:48:59] So yeah, maybe maybe an unlearning for
[00:49:01] me, a learning more concretely has been
[00:49:03] just to accept there’s a lot of
[00:49:05] uncertainty. We have to there’s a lot of
[00:49:07] different actors and it’s a democratic
[00:49:09] process. So even if I have a strong
[00:49:11] view, somebody else might have a strong
[00:49:13] view the opposite and both of our voices
[00:49:15] need to be heard by policy makers and
[00:49:17] you just can’t accept that whatever I
[00:49:19] say will just be accepted very easily.
[00:49:21] So, so maybe that’s that’s one thing
[00:49:22] that uh I realized through this uh
[00:49:24] process over the years and
[00:49:25] >> as I like to state it that change is the
[00:49:27] only constant. Any advice that you being
[00:49:30] at the forefront of policy would want to
[00:49:32] leave for our listeners like CTO’s and
[00:49:35] IT directors?
[00:49:36] >> I mean some of these tools could be
[00:49:37] useful for them that I already
[00:49:39] mentioned. Maybe I will also say that
[00:49:41] it’s useful to see this regulation as
[00:49:44] basically a tool that helps um the
[00:49:47] market work better, the EU markets and
[00:49:50] hopefully the global markets work better
[00:49:51] because if we have a framework in place,
[00:49:53] it can potentially lead to more legal
[00:49:56] certainty. We have an environment where
[00:49:57] we can operate with our business and we
[00:50:00] know what to expect potentially can be
[00:50:01] very useful for for innovation and for
[00:50:04] business. often times maybe
[00:50:05] entrepreneurs they’re more excited about
[00:50:07] other kinds of things like the rword
[00:50:09] regulation doesn’t sound like a very
[00:50:11] appealing thing and so often times
[00:50:13] people are like there have been surveys
[00:50:15] where people are asked I think there was
[00:50:17] a survey maybe by politico a while ago
[00:50:19] where investors and startups were asked
[00:50:22] hey which of these regulations are
[00:50:24] hindering your work and for example some
[00:50:26] of them said that the digital services
[00:50:28] act by the EU was like hindering their
[00:50:31] work I to myself thought it’s so so
[00:50:33] weird that all these startup would say
[00:50:34] that because the digital services act
[00:50:36] only applies to very large online
[00:50:38] platforms just like some of the biggest
[00:50:40] companies in the world. So no startup
[00:50:41] basically almost ever will have rules
[00:50:43] under like obligations under that. I was
[00:50:45] like why are you like so worried about
[00:50:47] this this law? My hypothesis was that
[00:50:50] they just whatever a law there is you
[00:50:52] you’re opposed to it because that’s not
[00:50:54] your game. Your game is investments
[00:50:57] building products and services and like
[00:50:59] that’s the interesting part and that’s
[00:51:00] all fine. That’s that’s your comparative
[00:51:02] advantage. But also see the other side
[00:51:04] that it can be helpful for innovation.
[00:51:06] If we have a nice legal framework, legal
[00:51:08] certainty, everybody operates under the
[00:51:10] same rules. When I use a model and it’s
[00:51:12] come through this evaluation, testing
[00:51:14] process, mitigation process, it could be
[00:51:16] more safe that helps me build out on top
[00:51:18] of it and I don’t get into incidents.
[00:51:20] Maybe those incidents could mean legal
[00:51:23] obligations on on the upstream, but
[00:51:25] perhaps under certain certain
[00:51:27] circumstances it could be on me and I
[00:51:28] don’t want that, right? So, so I I
[00:51:30] should be interested in yeah seeing also
[00:51:32] the benefits of uh of these frameworks.
[00:51:34] So that would be my advice like also see
[00:51:36] the innovation benefits of uh of certain
[00:51:38] kind of regulations at least good
[00:51:39] regulations or good parts of
[00:51:40] regulations.
[00:51:41] >> Well Resto thank you so much for your
[00:51:43] honest and candid conversation for our
[00:51:45] listeners today.
[00:51:46] >> Thank you so much for having me. [music]