Essay by Eric Worrall
I’m currently working on three AI projects. All of the projects could produce novel AI capabilities. Which project should I “pace”?
The first project is a Kaggle competition – Kagriculture. At the moment my position is between 6000-7000 on a leaderboard of 10099 entries. The competition only has a few days to run. The payoff is US $5000 for winning a top position on the leaderboard. I’m almost certainly not going to win this competition, but I haven’t given up yet. The winner will be expected to present details of how their optimisation approach helped them win. But optimisation problems have far wider application than a farming competition on Kaggle, they can be used for everything from automating production lines, improving corporate profits, stock trading algorithms or automating aspects of military planning, logistics and combat strategy. Should competitors “pace” this competition because of the small possibility one of them will stumble across a major breakthrough which helps push forward the capability of military Artificial Intelligence?
The second project is a computer vision project which will help substantially improve safety in an important industry. The vision project could save lives. There is a big payoff if I or anyone else can make it work. The client has provided some very challenging conditions, but over a decade ago I worked on a primitive computer vision project which faced a comparable problem, and found an unusual old technology solution to improving the interpretability of challenging visual environments.
If someone solves the problem, one day that solution could be used for a lot more than improving safety. An innovation which improves robot visual acuity in difficult circumstances could boost the capabilities of hunter seeker terminator robots looking for enemies of the state in difficult terrain like underground tunnels or other complex environments. Should researchers abandon this safety project, because of the possibility that one day their safety feature breakthrough might be used to hurt people?
The third project is for a University, its an interesting challenge. I’m not going to make money directly from this project, though in the long term it will help me secure a higher income. What if one of the people tackling this challenge thinks of an idea which leads to an important new advance? Should we “pace” this project, because of the remote possibility one of the solutions to the challenge pushes forward the capabilities of AI?
My point is AI innovation is a series of small incremental steps. There is almost never a red flag which says “you’ve gone too far, this is where you should pace your AI development” – there’s just another problem to solve, and a payoff at the end for your solution. Occasionally someone realises they have had a consequential insight, but most of the time, its just an ordinary Joe or Eric solving day to day work related problems. Someone who has a job to do and bills to pay.
What form would the proposed policing of AI take?
To have any chance of policing AI advances which even the inventors of those advances don’t necessarily know are important, big picture experts with a high level of knowledge of the state of the industry would have to be inserted into every company with the potential to make significant advances, no matter how small.
From Anthropic CEO Dario Amodei’s call for pacing the Advancement of AI;
…
Embedded Evaluators. Each frontier AI company commits to giving ongoing, employee-like access to a team of embedded third-party evaluators (such as METR), whose role is to verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes. This is the key step for verifiability of any pacing commitments, and has precedent in the banking industry, which sometimes involves regulatory “supervisors” embedded along with employees. Anthropic is unilaterally committing to this step now. We intend this to be part of a broader push to redouble efforts on our safety and alignment work.
…
Read more: https://darioamodei.com/post/we-must-pace-the-frontier
Top AI people with sufficient experience and expertise to make a call on any individual advance are not cheap. Paying for an “embedded evaluator” would wipe out the small end of the AI business, unless they perhaps negotiated a sponsorship and profit sharing arrangement with a large incumbent player.
There is an additional problem with this proposal. Questions have been raised about the alleged close ties between personnel in METR, Anthropic’s proposed AI oversight body, and AI giants like Anthropic.
…
The documented network connections
METR grew out of the Alignment Research Center, or ARC. The organizations are associated with Paul Christiano, who worked with Amodei at OpenAI in the 2010s and later served as one of the first five trustees of Anthropic’s Long-Term Benefit Trust.
The reported relationships also involve funding networks connected to Anthropic investors Dustin Moskovitz and Jaan Tallinn. The Survival and Flourishing Fund, associated with Tallinn, reportedly earmarked up to $752,000 for METR since its founding: $324,000 in general cash and a $428,000 matching pledge.
Coefficient Giving, associated with Moskovitz, reportedly gave $1,515,000to ARC in 2022. METR says that donation was firewalled and was not used for its operations.
Separately, METR announced approximately $71 million in funding commitments raised during the six months before its August 14 announcement. METR says it does not accept funding from AI companies or their employees and takes no compensation from AI labs.
The important distinction is between direct corporate funding and overlapping philanthropic or personal networks. The reported relationships concern the latter. METR’s stated policy concerns the former, as well as control over its projects.
…
Read more: https://www.neoteo.com/en/why-metrs-ties-are-raising-questions-about-anthropics-ai-watchdog-plan
I’m not accusing METR of deliberate collusion, I’m sure they are genuinely trying to be independent. But these people all know each other, and in many cases have worked together. In the high stakes world of artificial intelligence, it only takes one bad actor and a quick phone call, or even a careless comment at a lunch meetup, for a billion dollar idea to be leaked to people who have the resources to develop that idea faster than the author of that idea.
Imagine if inventors in other fields were forced to accept an “embedded evaluator”, who potentially has strong personal ties to big players who would profit enormously from stealing their ideas. I believe most people in the AI industry, in any industry, would find such a commercial risk intolerable, and would put a lot of effort into circumventing the requirement for such oversight.
Any attempt to police AI in the way proposed by some AI giants (not all) would be counterproductive – it would drive commercially sensitive AI research underground. In many cases researchers have the resources they need to conduct AI research at home, why should they tell anyone else what they are doing? Most AI advances don’t need to be tested on billion dollar computers, a $10,000 computer has plenty of AI capacity for a proof of concept test. In the face of a serious clampdown, secret AI breakthroughs could be laundered through third party countries which hadn’t signed up to the AI oversight system.
If a clumsy oversight mechanism like what Anthropic CEO Dario Amodei proposed was implemented, we would end up knowing less about what cutting edge AI researchers were doing, not more.
