在数据严重异常的时间段, 分析团队和他们依赖的业务流程需要重新调整他们的模型,现在就开始产生相关的见解, 同时也减轻了中断对未来表现的影响.

那么这在实践中是如何工作的呢? This perspective is designed to help data science practitioners work with business stakeholders to create process and standards for identifying, 对影响机器学习模型的政权变化做出反应,并对未来进行验证.

完美体育都太熟悉由与covid相关的转变——消费者行为的改变——造成的业务中断了, disruptions in production and 供应链s, 以及额外的监管压力, 举几个例子. 对机器学习(ML)模型的影响不那么明显,但同样重要. After all, ML通过利用大量数据来创造更快、更准确的见解,从而创造价值, 帮助数据科学家和分析领袖——以及依赖他们的产出的业务流程——建模风险, forecast market forces and guide strategic decision making.

\r\n

Although these ML models are designed to respond to changes, 当输入数据与它们最初训练的数据有很大差异时,它们也很脆弱,表现很差.

\r\n

数据科学从业者将这种输入数据的剧烈变化理解为一种制度变化, 在这种情况下,是由于市场力量和消费者行为的显著变化,这是ML模型的输入和目标. Classic examples of regime shifts in the field of macroeconomics, 例如, 经济周期的中断是否会导致深刻而不可预测的政治和经济后果, as well as financial ruptures that culminate in recessions. 而识别和应对政权转移的数学很重要, 数据驱动型企业首先需要了解其数据操作的优势和劣势. 然后,他们可以发挥自己的数学能力,继续推动与库存管理相关的业务成果, 资产经济学, 销售预测, 欺诈检测, 风险管理, 顾客行为与营销.

\r\n

以实现这种弹性, this paper constructs a set of adaptive guidelines to monitor, react and respond to changes that impact the quality of ML models. This is built upon alignment between data science teams and business stakeholders who both design the operational models and data-driven culture needed for value-add analytics operations. 这不仅关乎2020年,也关乎由此带来的不可避免的变化. Wise data and analytics teams understand the certain and ongoing degradation of models and the likelihood and large impact of future massive disruptions.

\r\n

成功的数据和分析团队正在构建跟踪的数据操作, 对永恒的变化和巨大的政权转变的影响作出反应和反击(Figure 1).

\r\n"}}' id="text-29c76182f0">

完美体育都太熟悉由与covid相关的转变——消费者行为的改变——造成的业务中断了, disruptions in production and 供应链s, 以及额外的监管压力, 举几个例子. 对机器学习(ML)模型的影响不那么明显,但同样重要. After all, ML通过利用大量数据来创造更快、更准确的见解,从而创造价值, 帮助数据科学家和分析领袖——以及依赖他们的产出的业务流程——建模风险, forecast market forces and guide strategic decision making.

Although these ML models are designed to respond to changes, 当输入数据与它们最初训练的数据有很大差异时,它们也很脆弱,表现很差.

数据科学从业者将这种输入数据的剧烈变化理解为一种制度变化, 在这种情况下,是由于市场力量和消费者行为的显著变化,这是ML模型的输入和目标. Classic examples of regime shifts in the field of macroeconomics, 例如, 经济周期的中断是否会导致深刻而不可预测的政治和经济后果, as well as financial ruptures that culminate in recessions. 而识别和应对政权转移的数学很重要, 数据驱动型企业首先需要了解其数据操作的优势和劣势. 然后,他们可以发挥自己的数学能力,继续推动与库存管理相关的业务成果, 资产经济学, 销售预测, 欺诈检测, 风险管理, 顾客行为与营销.

以实现这种弹性, this paper constructs a set of adaptive guidelines to monitor, react and respond to changes that impact the quality of ML models. This is built upon alignment between data science teams and business stakeholders who both design the operational models and data-driven culture needed for value-add analytics operations. 这不仅关乎2020年,也关乎由此带来的不可避免的变化. Wise data and analytics teams understand the certain and ongoing degradation of models and the likelihood and large impact of future massive disruptions.

成功的数据和分析团队正在构建跟踪的数据操作, 对永恒的变化和巨大的政权转变的影响作出反应和反击(Figure 1).

Figure 1. 数据价值链图可以确保数据操作在战略上与关键业务结果保持一致.

完美体育有责任了解源数据不可预见的中断的必然性, 以及如何保持指导战略和为行动提供信息的洞察力的连续性.

\r\n

最初的两个步骤至关重要:

\r\n
    \r\n
  • Understand exactly what will cause ML models to falter
  • \r\n
  • 建立详细的, 对不同程度的故障进行识别和反应的文件化和商定的计划
  • \r\n
\r\n

在危机时期, ML models do not outperform conventional statistical models, 因为它们是, by design, incapable of identifying biases in the data. 模型风险通常会随着不正确的假设和输入而增加, 这使得从大量数据中获取商业智能变得极其困难. Standard knowledge among DS practitioners is that both traditional and ML models’ forecasting power depends on the quality of the data being fed and the set of assumptions made regarding the underlying data-generating process.

\r\n

在危机期间,模型通过/失败阈值的突破比正常时期更多,这并不奇怪. In finance, 例如, 管理信用或贷款组合表现的过程在不同的制度中是不同的,这是一个被广泛记录的过程. 不能适应结构突变会导致更大的错误和糟糕的模型性能统计数据.

\r\n

健壮的建模方法需要与通过/失败阈值相关的详细标准. These thresholds should not be considered a simple KPI or alarm bell for the model, 但应该改为触发记录, 对表现不佳的模型的细微指标作出反应的详细和一致同意的过程.

\r\n

这里有三个提示,可以帮助您建立识别和沟通模型失败或性能下降的标准:

\r\n"}}' id="text-1d26476a12">

 

如何创建健壮的标准来应对中断和退化

Few, if any, 数据科学家 could have predicted the drastic increase in demand for toilet paper during 2020 and the ripple effect on consumer behavior and retail 供应链s. Regardless, 数据科学从业者处于对全球公司成功至关重要的价值链中, 完美体育有责任了解源数据不可预见的中断的必然性, 以及如何保持指导战略和为行动提供信息的洞察力的连续性.

最初的两个步骤至关重要:

  • Understand exactly what will cause ML models to falter
  • 建立详细的, 对不同程度的故障进行识别和反应的文件化和商定的计划

在危机时期, ML models do not outperform conventional statistical models, 因为它们是, by design, incapable of identifying biases in the data. 模型风险通常会随着不正确的假设和输入而增加, 这使得从大量数据中获取商业智能变得极其困难. Standard knowledge among DS practitioners is that both traditional and ML models’ forecasting power depends on the quality of the data being fed and the set of assumptions made regarding the underlying data-generating process.

在危机期间,模型通过/失败阈值的突破比正常时期更多,这并不奇怪. In finance, 例如, 管理信用或贷款组合表现的过程在不同的制度中是不同的,这是一个被广泛记录的过程. 不能适应结构突变会导致更大的错误和糟糕的模型性能统计数据.

健壮的建模方法需要与通过/失败阈值相关的详细标准. 这些阈值不应该被认为是模型的简单KPI或警报, 但应该改为触发记录, 对表现不佳的模型的细微指标作出反应的详细和一致同意的过程.

这里有三个提示,可以帮助您建立识别和沟通模型失败或性能下降的标准:

以有利于战略讨论的格式记录影响模型性能的因素. During the construction and operation of a model, 评估不同因素对模型预测能力的不同程度影响. 创建一个框架来描述每个因素对模型性能的影响. 使用该框架主动评估数学含义, but also the business implications of regime shift. Generate documentation and referenceable assets that data science practitioners and business stakeholders can use to discuss changes to input data and the resulting impact on model performance.

Create a communication plan for calling out the factors leading to larger errors and deterioration of the model’s predictive power. 轻微中断和完全中断的区别是什么? 与2个月的中断相比,2周的中断有多重要? Extend your framework for describing changes to the model, 建立触发器, 随着时间的推移,提高标准和应急计划以保持反应的一致性. 不应该让涉众以一种特殊的方式来解释中断的严重性, 但相反,应该能够从一个商定的参考和指定开始.

决定在何时以及如何对模型进行重新校准时应考虑哪些管理判断. Identify who is responsible for and dependent on the models, 谁对必须做出的各种战略和战术决定有最终决定权. 记录他们在发现过程中的角色, assessing and reporting disruption as well as in researching, 设计和承诺补救. Something as simple as a responsibility assignment matrix for reacting to different thresholds of disruption can be a worthwhile investment.

如果坏了,就修好它

识别中断的标准与对中断作出适当反应的标准是不同的. So how do you know when to make a change to the model?

Making changes to machine learning models is not simple. The pandemic has revealed how entangled our lives are with AI. 在行为变化和人工智能模型的工作方式之间存在着微妙的相互依赖关系, 因此,需要考虑大量关于数据和相关业务驱动因素的变量.

以下是决定何时以及如何对模型进行更改的三个关键技巧:

  1. 不要因为行动应该是“自动的”就害怕采取行动.” 密切关注那些勉强维持运转的自动化系统是很重要的. When you are faced with extraordinary circumstances, you cannot set models on auto pilot and be fully hands off. 在需要的时候,对自动化系统进行手动调整是至关重要的.
  1. 确保每个人在适应政权更迭时都知道自己在做什么. 数据科学从业者可用的方法范围从简单到复杂. 新的数学和计算机模型将作为潜在的解决方案出现. 对它们的评估不仅应看其改善模式性能的可行性, 但也考虑到变革的程度,管理和实施成本将需要采用它们. 花点时间确保您了解最新的行业最佳实践和新解决方案的实际用例细节.
  2. Use design best practices and be transparent about uncertainty. 你关注的中心是数据科学和机器学习,但不要忘记人. Understand your changes as more than just mathematics, and consider the needs from a variety of stakeholders. 利用设计思维最佳实践来确保您恰当地定义了问题空间. 当会议, 明确区分关于可能性的生成式讨论和方向对齐, compared to focused efforts to scope designs, refine solutions and move forward down a chosen path. 利用工程师的输入,P&L owners, users of analytics reports and a breadth of stakeholders. This will allow you to design the best solution to the disruption, 同时也要对问题和解决途径保持高度的透明度.

实际上,设计思维和数据分析能力是如何集成的? 

Future proof with resilient models and data operations

数据科学家花在对模型退化进行分类和做出反应上的时间越多, 他们花在用数据推动业务成果的创新方法上的时间就越少. It is possible to create models that hold up even when the environment changes — provided the ranges for the features remain broadly identical to what have been tested previously (Figure 2).

Figure 2. Sustainable value from ML comes from more than the models themselves. 确保你能看到数据表面下的连接,从而洞察价值链.

 

考虑以下方法,使当前和未来的模型更能适应破坏:

如果模型确定了制度的变化,就不再依赖历史数据. Since machine learning models are trained on data, 如果您不相信获取数据的环境与当前设置是可比较的, you should try simpler models that rely on fewer features or that deploy variables that explain the final output or result of the models. Reducing the number of features will prevent you from being misled by a questionable dataset and help you understand what will still work and what might not in the new environment.

为如何处理丢失或严重破坏的数据建立需求和最佳实践. 数据科学实践者在处理缺失数据时应该格外小心. 当制造业, 供应链, 工业和其他业务流程在被迫关闭期间停止, missing data can adversely impact the future training of ML models.

To alleviate future issues related to missing data, 保存事件日志,以及在此期间它们如何在数据中表示. In order to minimize time spent on future data cleansing, 记录这些数据问题, 只要有可能, 确保包含异常的原因和预期的潜在后果.

It is possible to overcome issues in data irregularities and limit the alteration to the original time series either by omitting the data or inputting it manually. 例如,考虑对模型再训练频率应用更短的时间框架. This is particularly effective for industries — financial markets, 例如,数据不规范引起的问题是短暂的. Don’t be too concerned about model performance in this scenario, 因为补救措施只是暂时的,不会影响整体表现.

Data from a previous crisis (usually economic) may appear to be the most likely candidate to inform management or the model developers about the overlays. However, 需要注意的是,数据生成机制在危机期间可能会有很大差异, 因此,依赖以往的危机数据可能是一个糟糕的未来基准.

更频繁地再培训,如果统计上合适的话,使用更小的数据集. When faced with a temporal component in the data, models in multiple business applications should be re-trained frequently with smaller data sets if it is statistically accommodatable and economically sensible. The purpose is to become more agile and better capture the quickly changing nature of the real-world socioeconomic dimension they model.

除了, 以下最佳实践有助于创建不太容易发生灾难性故障的ML模型:

尽可能集成更多的数据源,增加数据收集的频率和粒度. 将数据收集的频率和粒度从每周增加到每天, 或者每天到每小时, 例如, 能否帮助缩短学习周期并提高对数据段变化的理解.

In the case of financial markets forecasting, 例如, 模特们不应该只接受过去几年的跌宕起伏的训练, 也包括黑天鹅事件,比如1987年的“黑色星期一”股市崩盘, 2000年互联网泡沫破裂,2007-2008年全球金融危机爆发. 当然, 像冠状病毒这样的健康危机是建立更好的ML模型的完美触发器.

Create a log of events and how they are represented in the data, and document reasons and expected consequences. ……很重要。 for 数据科学家 to resist the temptation to throw 2020 data out of their forecasting models entirely due to anomalies and instead rely on 2019 and 2018 data.

诚然,政府和监管机构正在实施一些生硬的工具来启动经济, 造成消费者行为异常. However, 这些异常现象不应被忽视, and you should train your models to guide you through them, 在必要时加入虚拟变量来捕获和解释异常期.

尽管经济不太可能在短期内面临如此严重的全球影响, we more frequently do encounter regime shifts in regional context. 例如, 由于恶劣天气或自然灾害造成的关闭可能会对当地产生重大影响,而且更为常见. ……很重要。, therefore, 通过异常(或异常情况)训练模型,以理解影响, duration and recovery rate of the business, so that you can make smarter decisions at scale and in less time.

承认偏见的存在. Biases can arise from ML models when they are tasked with implementing policy decisions as a result of unrepresentative training data sets. 虽然这不是本文要进一步阐述的主题, 探索ML算法中偏差的预期和意外后果是很重要的.

For example, 当用于训练模型的输入数据基于性别更能代表特定人群时, 种族和其他因素, 该模型的预测也可能系统地偏向于这些群体. Hence, these biases can lead to decisions that can adversely affect specific groups of people without the developer explicitly modeling them.

完美体育博客) 

\r\n"}}' id="text-0e20ae3d52">

创建一个更敏捷的实践——MLOps. 引用塞巴斯蒂安的话Klöser, 解决方案主要, Luxoft的人工智能和数据科学, 完美体育科技公司, “Most companies have focused on building machine learning muscle — hiring 数据科学家 to create and apply algorithms capable of extracting insights from data. This makes sense, but it’s a rather limited approach. Think of it this way: They’ve built up the spectacular biceps but haven’t paid as much attention to the underlying connective tissues that support the muscle.” (Source: Elevate AI development by applying MLOps principles, Sebastian Klöser, 完美体育博客) 

MLOps就是结缔组织. It integrates machine learning with software development best practices to ensure businesses get real and continuous value from machine learning and artificial intelligence applications. 这绝不能掩盖数据科学家展示他们力量的能力. 正如克洛泽所观察到的, “数据科学家可以继续使用他们喜欢的工具和方法, 它们的输出由松散耦合的DevOps和DataOps接口提供. 他们的ML算法开发工作成为高度专业的工厂系统的核心, 可以这么说.”

其他人是怎么做的?

Clearly, 各种因素的复杂性影响着机器学习模型及其与企业环境的集成. 作为最佳实践, 始终保持对组织之外的更改的处理方式的最新了解是值得的.

当检查合作伙伴, platforms, 解决方案和技术, 重要的是,不仅要考虑他们方法的数学有效性, but also their relationship to problem solving. 他们和你面临同样的问题吗? Did they have the same resources and intended outcome? 核心, we are trying to solve problems through machine learning, 不仅仅是产生漂亮的数学.

一种传统的计量经济学方法用于制度转移是马尔可夫转换模型. 在这个政权转换模型中, 政权之间的变化是根据一个未观察到的马尔科夫链演变的, 根据一定的概率规则从一种状态转换到另一种状态的数学系统.

A regime-switching model is a parametric model of a time series in which parameters are allowed to take on different values in each of some fixed number of regimes. 该方法被计量经济学家和定量分析师广泛应用于金融市场建模, 并广泛应用于时间序列数据.

Despite attempts to recalibrate models with traditional approaches, 随着新的数据体制的出现,以前已经建立的模型陷入困境是正常的. 可选择的建模方法可以作为可靠的机制或临时解决方案, 甚至提供一个利用从颠覆中出现的新趋势和模式的机会.

Augmenting supervised machine learning techniques with alternative techniques such as reinforcement learning can help businesses rapidly adapt to new economic behavior. 这种机器学习技术使智能体能够通过试错在交互式环境中学习, using feedback from its own actions and experiences. 强化学习使用奖励和惩罚作为积极和消极行为的信号.

另一种非常适合于主要制度变化的建模方法是贝叶斯建模. This technique has intrinsic abilities to measure uncertainty, 包括先验知识和嵌入变量之间的结构关系. Bayesian networks also support causally grounded counterfactual analysis to infer what happens to a target (dependent) variable when an intervention is performed on a feature (predictor) variable.

其他较强的候选成功建模相对较短的数据集是层次线性模型(HLM), Gaussian processes and other probabilistic graphical models (PGM).

结论:完美体育在一起

如果这算安慰的话, 你可以肯定,你在私人和公共组织中的同行也在处理与你相同的干扰. As processes are “digitalized” and data is “democratized,” we are having less trouble accessing data, 更麻烦的是确保它干净,然后弄清楚如何处理所有这些.

完美体育敲响了警钟.

\r\n

现在是时候对这些新暴露的系统进行评估,并询问如何更好地设计它们,使它们更具弹性. If machines are to be trusted, we need to closely monitor them. 一组经验丰富的数据科学家可以帮助您将世界上正在发生的事情与算法正在发生的事情联系起来, 并引入操作最佳实践和经过深思熟虑的DataOps计划.

\r\n

这将使ML在现在和可预测的不可预测的未来都能最好地服务于业务.

\r\n

关于数据分析中的完美体育
\r\n

\r\n

完美体育技术 has a cross-functional team of consultants, 数据科学家, designers, 帮助客户实现高级企业数据解决方案的架构师和工程师. 完美体育的数据和分析团队可以通过建立大规模的模型治理来帮助您成功地导航中断, enhancing collaboration between data and business teams, and deploying the foundation for resilient MLOps. 学习更多在 idqhracing.com/data-analytics.

\r\n

 

\r\n"}}' id="text-03b7c7472a">

如果您决定投资于一个健壮而又具有适应性的数据科学实践的基础, 你已经走在潮流的前面了. But beware of one pitfall: Most of the challenges businesses encounter with their ML models are because an increasing number of them are buying off-the-shelf ML systems and lack the in-house know-how needed to maintain and retrain them. 只有少数企业拥有将复杂的替代方法付诸实践的技能和数据基础设施.

对于那些假设所有自动化ML模型都可以自己运行的企业, 目前的形势给完美体育敲响了警钟.

现在是时候对这些新暴露的系统进行评估,并询问如何更好地设计它们,使它们更具弹性. If machines are to be trusted, we need to closely monitor them. 一组经验丰富的数据科学家可以帮助您将世界上正在发生的事情与算法正在发生的事情联系起来, 并引入操作最佳实践和经过深思熟虑的DataOps计划.

这将使ML在现在和可预测的不可预测的未来都能最好地服务于业务.

关于数据分析中的完美体育

完美体育技术 has a cross-functional team of consultants, 数据科学家, designers, 帮助客户实现高级企业数据解决方案的架构师和工程师. 完美体育的数据和分析团队可以通过建立大规模的模型治理来帮助您成功地导航中断, enhancing collaboration between data and business teams, and deploying the foundation for resilient MLOps. 学习更多在 idqhracing.com/data-analytics.

 

完美体育技术的分析和工程高级数据科学家吗. He has extensive experience in quantamental research and investing, having worked for global investment banks and asset management, 对冲基金和科技公司. He has designed and delivered leading analytics solutions by discovering exploitable patterns in the data to realize profitable outcomes, 并通过实施基于高级统计分析的系统策略来帮助提高绩效, ML, 跨结构化和非结构化数据的数据挖掘和数据可视化技术.

\r\n"}}' id="text-6f28367224">

关于作者

拉吉夫Thillainathan 是亚太地区完美体育技术的分析和工程高级数据科学家吗. He has extensive experience in quantamental research and investing, having worked for global investment banks and asset management, 对冲基金和科技公司. He has designed and delivered leading analytics solutions by discovering exploitable patterns in the data to realize profitable outcomes, 并通过实施基于高级统计分析的系统策略来帮助提高绩效, ML, 跨结构化和非结构化数据的数据挖掘和数据可视化技术.