2020年更新和战略

||Miri战略,News

米里的2020年是一个试验和调整的一年。为了应对COVID-19大流行,我们在3月份基本上将行动转移到了更多的农村地区,并将重点转向了偏远地区的工作。我们借此机会尝试了新的工作设置和研究方法,并对结果非常满意。亚博体育官网

与此同时,到了2020年,米里领导层此前最为兴奋的研究进展有限:新的亚博体育官网亚博体育官网研究方向我们在2017年开始。鉴于我们迄今为止的进展缓慢,我们正在考虑对我们的战略进行一些可能的变化,而Miri的研究领导力正在转向他们对寻求更有前途的道路的大部分重点。亚博体育官网

去年,我预计我们2020年的预算将为640万美元至740万美元,预计为680万美元。我现在预计,我们2020年的支出将略高于740万美元。支出的增加超过了我的估计,这主要是因为我们为应对COVID-19大流行而重新安置工作人员和采取预防措施的支出。

我们的2021年预算相当不确定,因为我们比平常更有可能在来年看到我们的战略中的高级转变。我目前的估计是,我们的支出将落在600万美元至7.5米之间,我希望大致崩溃如下:

I’m also happy to announce that the Survival and Flourishing Fund (SFF)has awarded MIRI $563,000to support our research going forward, on top of support they provided今年早些时候.

考虑到我们的研究项目正处于过渡亚博体育官网期,考虑到我们今年已经得到了438万美元的大力支持来自公开慈善事业,$903k from SFF, and ~$1.1M from other contributors (thank you all!)—we aren’t holding a formal fundraiser this winter. Donations are still welcome and appreciated during this transition; but we’ll wait to make our case to donors when our plans are more solid. For now, see ouryabo体育官网下载ios 如果您对支持我们的研究感兴趣,请呼叫。亚博体育官网

下面,我将更详细地介绍我们的2020年是如何过去的,以及我们对未来的计划。

2017-Initiated Research Directions and Research Plans

2017年,我们介绍一套新的研究方向,我们在亚博体育官网2018年更新:我们的新研究方向亚博体育官网.” We wrote that we were “seeking entirely new low-level foundations for optimization,” “endeavoring to figure out parts of cognition that can be very transparent as cognition,” and “experimenting with some specific alignment problems.” In December 2019, we noted that we felt we were making “steady progress” on this research, but weredisappointed with the concrete results we’d had to date.

在推动这些研究领域的进一步发展之后,MIRI的高级职员对这种方法变得更加悲亚博体育官网观。MIRI执行董事兼高级研究员Nate Soares写道:

脸上的非公开研究最令人兴奋的是,尝试为可对准亚博体育官网的AI开发新的典型可行基础,这并没有通过梯度 - 下降式机器学习基础来依赖于路由。尽管有明显的困难,我们有各种各样的理由希望这可以工作。

That project has, at this point, largely failed, in the sense that neither Eliezer nor I have sufficient hope in it for us to continue focusing our main efforts there. I’m uncertain whether it failed due to implementation failures on our part, due to the inherent difficulty of the domain, or due to flaws in the underlying theory.

鉴于我们的距离可能的距离和对准问题的难度感,我们失去希望的一部分是我们失去希望的感觉太慢了。AI对准领域正在截止日期下工作,使得如果工作进展缓慢,我们会更好放弃and pivoting to new projects that have a real chance of resulting in the first AGI systems being built on alignable foundations.

We are currently in a state of regrouping, weighing our options, and searching for plans that we believe may yet have a shot at working.

看着整个领域,Miri的研究领导力仍然非常悲观,这对我们所看到的大多数一致性提案亚博体育官网迄今为止提出。也就是说,我们对近期研究方向更加悲观的更新并没有减少我们对替代方面的悲观主义,我们承担的下一个方向不太可能类似于今天Miri外的流行方亚博体育官网向。

Miri认为需要改变这些项目的课程。与此同时,许多(包括Nate)仍然存在于本研究的理论中的一些希望,并希望这些项目可能会以某种方式救出,例如通过发现和纠正如何在我们接近这项研究中的失败。亚博体育官网但是在救援工作上花费了努力,反对找到更好,更有前途的对齐计划。

所以我们做一些改变影响员工的公关eviously focused on this work. Some are departing MIRI for different work, as we shift direction away from lines they were particularly suited for. Some are seeking to rescue the 2017-initiated lines of research. Some are pivoting to different experiments and exploration.

我们不确定我们将决定的长期计划,并在生成新可能的策略过程中。一些(非相互独家)可能性包括:

  • 我们可能成为各种研究方法的家庭,旨在开发一个以开发对准的新道路。亚博体育官网鉴于我们对最佳攻击角度增加的不确定性,它可能是有价值的,以便储备更多样化的项目组合,在方法之间具有一定程度的互通和交叉授粉。
  • 如果我们能够确定我们认为有机会从AGI确保积极成果的真正可能性,我们可能会犯下完全新的方法。
  • 我们可以将2017年启动的研究方向中的理论和见解以不同的形式推广到未来的计划中。亚博体育官网

亚博体育官网研究写作

Although our 2017-initiated research directions have been our largest focus over the last few years, we’ve been running many other research programs in parallel with it.

这项工作的大部分是默认不公开as well, but it includes work we’ve written up publicly. (Note that as a rule, this public-facing work is unrepresentative of our research as a whole.)

从我们的角度来看,今年我们最有趣的公共工作是Scott Garrabrant的笛卡尔框架模型和Vanessa Kosoy的基础贝叶斯理论。

笛卡尔坐标系是一个关于代理商的新框架,旨在作为一个继任者cybernetic agent model. 尽管控制论agent模型假设一个agent和环境是一个基本的,它在时间上具有一个定义的和稳定的I/O通道,笛卡尔框架将这些特性视为更衍生的,并且依赖于一个人在概念上如何划分物理环境。

The Cartesian Frames sequence focuses especially on finding derived, approximation-friendly versions of the notion of “subagent” (previously discussed in “嵌入式代理“)和时间序列(一个源decision-theoretic problemsin cases where agents can base their decisions on predictions or proofs about their own actions). The sequence’s final post discusses these and other potential今后工作方向for the field to build on.

一般来说,美里的研究人员非常inter亚博体育官网ested in new conceptual frameworks like these, as research progress can often be bottlenecked on our using the wrong lenses for thinking about problems, or on our lack of a simple formalism for putting intuitions to the test.

与此同时,Vanessa Kosoy和Alex Appel的红贝叶岛主义是一种用于在推理的假设空间可能不包括真实环境的情况下建模推理的新框架。

该框架主要是有趣的,因为它似乎适用于如此多的问题:不可实现,决策理论,人类学,嵌入式代理,反射和诱导/逻辑的诱导/概率的合成。凡妮莎将红外贝叶斯主义描述为“向往来将学习理论应用于许多似乎与之似乎不相容的问题的方式开辟道路。”

2020年也出现了大更新到斯科特和亚伯兰的“嵌入式代理,并澄清了一些讨论,增加了几个新的小节。此外,瓦妮莎的最优多项式时间估计:逼近算法的贝叶斯概念,” co-authored with Alex Appel, was published in the应用逻辑杂志.

为了展示我们一直在推动的其他一些研究领域的照片,我们向一些Miri研究人员和研亚博体育官网究员们询问了过去一年中的工作中的亮点,评论了他们的选择。

艾布拉姆·德姆斯基强调了以下评论:

Evan Hubinger summarizes his public research from the past year:

Earlier this year, Buck Shlegeris (link)还有埃文·胡宾格(link) also appeared on the Future of Life Institute’s AI Alignment Podcast. Buck also gave a talk at Stanford: “我个人在人工智能安全方面的工作重点.”

最后,人类未来研究所研究员和MIRI研究助理Stuart Armstron亚博体育官网g总结了自己的研究重点:

  • 在线学习奖励函数的陷阱,” working with DeepMind’s Jan Leike, Laurent Orseau, and Shane Legg — “This shows how agents can manipulate a “learning” process, the conditions that make that learning actually uninfluenceable, and some methods for turning influenceable learning processes into uninfluenceable ones.”
  • 模型分裂“-”在这里,我认为许多人工智能的安全问题可以归结为同一个问题:处理当你从训练数据中移出分布时发生的事情。我认为,要获得安全的人工智能,必须有一种处理这些“模型碎片”的原则性方法,并勾勒出一些例子。”
  • 语法,语义和符号接地,简化“ - ”在这里,我认为象征接地是一种实用,必要的事情,而不是抽象的哲学概念。“

过程改进和计划

Given the unusual circumstances brought on by the COVID-19 pandemic, in 2020 MIRI decided to run various experiments to see if we could improve our researchers’ productivity while our Berkeley office was unavailable. In the process, a sizable subset of our research team has found good modifications to our work environment that we aim to maintain and expand on.

Many of our research staff who spent this year in live-work quarantine groups in relatively rural areas in response to the COVID-19 pandemic have found surprisingly large benefits from living in a quieter, lower-density area together with a number of other researchers. Coordination and research have felt faster at a meta-level, with shorter feedback cycles, more efforts on more cruxy experiments, and more resulting pivots. Our biggest such pivot has been away from our 2017-initiated research directions, as described above.

Separately, MIRI staff have been weighing the costs and benefits of possibly moving somewhere outside the Bay Area for several years—taking into account the housing crisis and other governance failures, advantages and disadvantages of the local culture, tail risks of things taking a turn for the worse in the future, and otherfactors.

部分原因是这些考虑因素,部分是因为当我们今年的许多人因为Covid-19已经重新安置时,Miri正在考虑迁离Berkeley,因此更容易移动。当我们权衡选项时,我们考虑的特别是大量因素是我们的研究人员是否期望地理位置,生活情况和工作设置感觉良好和舒适,因为我们通常期望这导致改进的研究进展。亚博体育官网越来越多地,这个因素指向我们走向一些新的新东西。

MIRI的许多人在过去注意到,有一些特定的社会环境,比如小型有效利他主义或结盟研究静修,似乎会引发异常高密度的异常富有成效的对话。这类静修活动的活力和活力很大程度上可能源于他们的新奇和受时间限制的天性。然而,我们怀疑这并不是这些活动趋于密集和富有成亚博体育官网效的唯一原因,我们相信,我们也许能够创造一个每天都有这些特点的空间。

This year, a number of our researchers have indeed felt that our new work set-up during the pandemic has a lot of this quality. We’re therefore very eager to see if we can modify MIRI as a workplace so as to keep this feature around, or further augment it.

Our year, then, has been characterized by some significant shifts in our thinking about research practices and which research directions are most promising.

尽管我们对最近在理解如何调整AGI等级优化方面取得的具体进展感到失望,但我们计划继续利用MIRI强大的人才库和积累的调整思路,寻找新的更好的前进道路。随着我们计划的巩固,我们将提供更多关于我们新战略的更新。