当前位置：首页 > news >正文

房屋中介做网站的书籍北京seo顾问服务

news 2025/8/4 5:26:21

房屋中介做网站的书籍,北京seo顾问服务,百度搜索链接入口,怎样做自己的微商网站6概述序列标注指给定输入序列，给序列中每个Token进行标注标签的过程。序列标注问题通常用于从文本中进行信息抽取，包括分词(Word Segmentation)、词性标注(Position Tagging)、命名实体识别(Named Entity Recognition, NER)等。条件随机场&#xff08…

概述

序列标注指给定输入序列，给序列中每个Token进行标注标签的过程。序列标注问题通常用于从文本中进行信息抽取，包括分词(Word Segmentation)、词性标注(Position Tagging)、命名实体识别(Named Entity Recognition, NER)等。

条件随机场（CRF）

对序列进行标注，实际上是对序列中每个Token进行标签预测，可以直接视作简单的多分类问题。但是序列标注不仅仅需要对单个Token进行分类预测，同时相邻Token直接有关联关系。

设 $x=\begin{Bmatrix} x_0,...,x_n \end{Bmatrix}$ 为输入序列， $y=\begin{Bmatrix} y_0,...,y_n \end{Bmatrix}$ 为输出的标注序列，输出序列y的概率为：

$P(y|x)=\frac{exp(Score(x,y))}{\sum_{y'\in Y}exp(Score(x,y'))}$

定义两个概率函数

1. 发射概率函数 $\psi _{EMIT}$ ：表示 $x_i\rightarrow y_i$ 的概率

2. 转移概率函数 $\psi _{TRANS}$ ：表示 $y_{i-1}\rightarrow y_i$ 的概率

于是可以得到Score的计算公式：

$Score(x,y)=\sum_ilog\psi _{EMIT}(x_i\rightarrow y_i)+log\psi_{TRANS}(y_{i-1}\rightarrow y_i)$

设标签集合为T，构造大小为 $\left | T \right |\times \left | T \right |$ 的矩阵P，用于存储标签间的转移概率。

实现CRF层的前向训练部分，将CRF和损失函数做合并，选择分类问题常用的负对数似然函数，则有：

$Loss=-log(P(y|x))$

$Loss=-log(\frac{exp(Score(x,y))}{\sum_{y'\in Y}exp(Score(x,y'))}) \newline=log(\sum_{y'\in Y}exp(Score(x,y'))-Score(x,y))$

Score计算

def compute_score(emissions, tags, seq_ends, mask, trans, start_trans, end_trans):# emissions: (seq_length, batch_size, num_tags)# tags: (seq_length, batch_size)# mask: (seq_length, batch_size)seq_length, batch_size = tags.shapemask = mask.astype(emissions.dtype)# 将score设置为初始转移概率# shape: (batch_size,)score = start_trans[tags[0]]# score += 第一次发射概率# shape: (batch_size,)score += emissions[0, mnp.arange(batch_size), tags[0]]for i in range(1, seq_length):# 标签由i-1转移至i的转移概率（当mask == 1时有效）# shape: (batch_size,)score += trans[tags[i - 1], tags[i]] * mask[i]# 预测tags[i]的发射概率（当mask == 1时有效）# shape: (batch_size,)score += emissions[i, mnp.arange(batch_size), tags[i]] * mask[i]# 结束转移# shape: (batch_size,)last_tags = tags[seq_ends, mnp.arange(batch_size)]# score += 结束转移概率# shape: (batch_size,)score += end_trans[last_tags]return score

Normalizer计算

Normalizer可以改写为以下形式：

$log(\sum_{y'_{0,i}\in Y}exp(Score_i))=log(\sum_{y'_{0,i-1}\in Y}exp(Score_{i-1}+h_i+P))$

Normalizer代码实现如下：

def compute_normalizer(emissions, mask, trans, start_trans, end_trans):# emissions: (seq_length, batch_size, num_tags)# mask: (seq_length, batch_size)seq_length = emissions.shape[0]# 将score设置为初始转移概率，并加上第一次发射概率# shape: (batch_size, num_tags)score = start_trans + emissions[0]for i in range(1, seq_length):# 扩展score的维度用于总score的计算# shape: (batch_size, num_tags, 1)broadcast_score = score.expand_dims(2)# 扩展emission的维度用于总score的计算# shape: (batch_size, 1, num_tags)broadcast_emissions = emissions[i].expand_dims(1)# 根据公式(7)，计算score_i# 此时broadcast_score是由第0个到当前Token所有可能路径# 对应score的log_sum_exp# shape: (batch_size, num_tags, num_tags)next_score = broadcast_score + trans + broadcast_emissions# 对score_i做log_sum_exp运算，用于下一个Token的score计算# shape: (batch_size, num_tags)next_score = ops.logsumexp(next_score, axis=1)# 当mask == 1时，score才会变化# shape: (batch_size, num_tags)score = mnp.where(mask[i].expand_dims(1), next_score, score)# 最后加结束转移概率# shape: (batch_size, num_tags)score += end_trans# 对所有可能的路径得分求log_sum_exp# shape: (batch_size,)return ops.logsumexp(score, axis=1)

Viterbi算法

在完成前向训练部分后，需要实现解码部分。Viterbi算法与计算Normalizer类似，使用动态规划求解所有可能的预测序列得分。不同的是在解码时同时需要将第i个Token对应的score取值最大的标签保存，供后续使用Viterbi算法求解最优预测序列使用。

取得最大概率得分ScoreScore，以及每个Token对应的标签历史HistoryHistory后，根据Viterbi算法可以得到公式：

$P_{0,i}=max(P_{0,i-1})+P_{i-1,i}$

代码实现：

def viterbi_decode(emissions, mask, trans, start_trans, end_trans):# emissions: (seq_length, batch_size, num_tags)# mask: (seq_length, batch_size)seq_length = mask.shape[0]score = start_trans + emissions[0]history = ()for i in range(1, seq_length):broadcast_score = score.expand_dims(2)broadcast_emission = emissions[i].expand_dims(1)next_score = broadcast_score + trans + broadcast_emission# 求当前Token对应score取值最大的标签，并保存indices = next_score.argmax(axis=1)history += (indices,)next_score = next_score.max(axis=1)score = mnp.where(mask[i].expand_dims(1), next_score, score)score += end_transreturn score, historydef post_decode(score, history, seq_length):# 使用Score和History计算最佳预测序列batch_size = seq_length.shape[0]seq_ends = seq_length - 1# shape: (batch_size,)best_tags_list = []# 依次对一个Batch中每个样例进行解码for idx in range(batch_size):# 查找使最后一个Token对应的预测概率最大的标签，# 并将其添加至最佳预测序列存储的列表中best_last_tag = score[idx].argmax(axis=0)best_tags = [int(best_last_tag.asnumpy())]# 重复查找每个Token对应的预测概率最大的标签，加入列表for hist in reversed(history[:seq_ends[idx]]):best_last_tag = hist[idx][best_tags[-1]]best_tags.append(int(best_last_tag.asnumpy()))# 将逆序求解的序列标签重置为正序best_tags.reverse()best_tags_list.append(best_tags)return best_tags_list

CRF层

CRF的输入需要考虑输入序列的真实长度，因此除发射矩阵和标签外，加入 seq_length 参数传入序列Padding前的长度，并实现生成mask矩阵的 sequence_mask 方法。

代码实现：

import mindspore as ms
import mindspore.nn as nn
import mindspore.ops as ops
import mindspore.numpy as mnp
from mindspore.common.initializer import initializer, Uniformdef sequence_mask(seq_length, max_length, batch_first=False):"""根据序列实际长度和最大长度生成mask矩阵"""range_vector = mnp.arange(0, max_length, 1, seq_length.dtype)result = range_vector < seq_length.view(seq_length.shape + (1,))if batch_first:return result.astype(ms.int64)return result.astype(ms.int64).swapaxes(0, 1)class CRF(nn.Cell):def __init__(self, num_tags: int, batch_first: bool = False, reduction: str = 'sum') -> None:if num_tags <= 0:raise ValueError(f'invalid number of tags: {num_tags}')super().__init__()if reduction not in ('none', 'sum', 'mean', 'token_mean'):raise ValueError(f'invalid reduction: {reduction}')self.num_tags = num_tagsself.batch_first = batch_firstself.reduction = reductionself.start_transitions = ms.Parameter(initializer(Uniform(0.1), (num_tags,)), name='start_transitions')self.end_transitions = ms.Parameter(initializer(Uniform(0.1), (num_tags,)), name='end_transitions')self.transitions = ms.Parameter(initializer(Uniform(0.1), (num_tags, num_tags)), name='transitions')def construct(self, emissions, tags=None, seq_length=None):if tags is None:return self._decode(emissions, seq_length)return self._forward(emissions, tags, seq_length)def _forward(self, emissions, tags=None, seq_length=None):if self.batch_first:batch_size, max_length = tags.shapeemissions = emissions.swapaxes(0, 1)tags = tags.swapaxes(0, 1)else:max_length, batch_size = tags.shapeif seq_length is None:seq_length = mnp.full((batch_size,), max_length, ms.int64)mask = sequence_mask(seq_length, max_length)# shape: (batch_size,)numerator = compute_score(emissions, tags, seq_length-1, mask, self.transitions, self.start_transitions, self.end_transitions)# shape: (batch_size,)denominator = compute_normalizer(emissions, mask, self.transitions, self.start_transitions, self.end_transitions)# shape: (batch_size,)llh = denominator - numeratorif self.reduction == 'none':return llhif self.reduction == 'sum':return llh.sum()if self.reduction == 'mean':return llh.mean()return llh.sum() / mask.astype(emissions.dtype).sum()def _decode(self, emissions, seq_length=None):if self.batch_first:batch_size, max_length = emissions.shape[:2]emissions = emissions.swapaxes(0, 1)else:batch_size, max_length = emissions.shape[:2]if seq_length is None:seq_length = mnp.full((batch_size,), max_length, ms.int64)mask = sequence_mask(seq_length, max_length)return viterbi_decode(emissions, mask, self.transitions, self.start_transitions, self.end_transitions)

BiLSTM+CRF模型

其中LSTM提取序列特征，经过Dense层变换获得发射概率矩阵，最后送入CRF层。具体实现如下：

class BiLSTM_CRF(nn.Cell):def __init__(self, vocab_size, embedding_dim, hidden_dim, num_tags, padding_idx=0):super().__init__()self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx=padding_idx)self.lstm = nn.LSTM(embedding_dim, hidden_dim // 2, bidirectional=True, batch_first=True)self.hidden2tag = nn.Dense(hidden_dim, num_tags, 'he_uniform')self.crf = CRF(num_tags, batch_first=True)def construct(self, inputs, seq_length, tags=None):embeds = self.embedding(inputs)outputs, _ = self.lstm(embeds, seq_length=seq_length)feats = self.hidden2tag(outputs)crf_outs = self.crf(feats, tags, seq_length)return crf_outs

完成模型设计后，我们生成两句例子和对应的标签，并构造词表和标签表。

embedding_dim = 16
hidden_dim = 32training_data = [("清 华 大 学 坐 落 于 首 都 北 京".split(),"B I I I O O O O O B I".split()
), ("重 庆 是 一 个 魔 幻 城 市".split(),"B I O O O O O O O".split()
)]word_to_idx = {}
word_to_idx['<pad>'] = 0
for sentence, tags in training_data:for word in sentence:if word not in word_to_idx:word_to_idx[word] = len(word_to_idx)tag_to_idx = {"B": 0, "I": 1, "O": 2}

接下来实例化模型，选择优化器并将模型和优化器送入Wrapper。

model = BiLSTM_CRF(len(word_to_idx), embedding_dim, hidden_dim, len(tag_to_idx))
optimizer = nn.SGD(model.trainable_params(), learning_rate=0.01, weight_decay=1e-4)grad_fn = ms.value_and_grad(model, None, optimizer.parameters)def train_step(data, seq_length, label):loss, grads = grad_fn(data, seq_length, label)optimizer(grads)return loss

将生成的数据打包成Batch，按照序列最大长度，对长度不足的序列进行填充，分别返回输入序列、输出标签和序列长度构成的Tensor。

def prepare_sequence(seqs, word_to_idx, tag_to_idx):seq_outputs, label_outputs, seq_length = [], [], []max_len = max([len(i[0]) for i in seqs])for seq, tag in seqs:seq_length.append(len(seq))idxs = [word_to_idx[w] for w in seq]labels = [tag_to_idx[t] for t in tag]idxs.extend([word_to_idx['<pad>'] for i in range(max_len - len(seq))])labels.extend([tag_to_idx['O'] for i in range(max_len - len(seq))])seq_outputs.append(idxs)label_outputs.append(labels)return ms.Tensor(seq_outputs, ms.int64), \ms.Tensor(label_outputs, ms.int64), \ms.Tensor(seq_length, ms.int64)

对模型进行预编译后，训练500个step。

from tqdm import tqdmsteps = 500
with tqdm(total=steps) as t:for i in range(steps):loss = train_step(data, seq_length, label)t.set_postfix(loss=loss)t.update(1)

最后将预测的index序列转换为标签序列，打印输出结果，查看效果。

idx_to_tag = {idx: tag for tag, idx in tag_to_idx.items()}def sequence_to_tag(sequences, idx_to_tag):outputs = []for seq in sequences:outputs.append([idx_to_tag[i] for i in seq])return outputssequence_to_tag(predict, idx_to_tag)

得到输出标签

[['B', 'I', 'I', 'I', 'O', 'O', 'O', 'O', 'O', 'B', 'I'],['B', 'I', 'O', 'O', 'O', 'O', 'O', 'O', 'O']]

总结

LSTM用于提取序列特征，CRF用于序列标注，从而实现语义的切分。

查看全文

http://www.ds6.com.cn/news/19405.html

网站制作企业网站建设哪家好线上推广员是做什么的

做网站有一个火箭回顶部新闻发稿

手机网站设计方案百度公司在哪里

买汽车最好的网站建设真正免费的网站建站平台有哪些

免费建网站视频教程关键词林俊杰mp3在线听

在源码之家下载的网站模板可以作为自己的网站吗湖南网站网络推广哪家奿

深圳最新疫情出行政策湖南seo博客seo交流

怎样安装微信小程序东莞市网站seo内容优化

网站优化就是每天更新内容吗爱站网长尾关键词挖掘工具的作用

做私彩网站代理会坐牢吗域名查询万网

概述