AVS开发相关知识
AVS相关知识
-
概念
Alexa Voice Service使用户获得基于云的Alexa能力，即将Alexa整合到产品中。通过在云端进行语音识别和自然语言理解，使构建语音产品更加简单
-
交互模型（Interactive model）
AVS由多个相应的客户端功能的接口(关于接口，下面有详细解释)组成，比如语音识别、音频播放和音量控制。每个接口包含逻辑上分组的消息，这些消息称为指令和事件。
指令是从云端发送的消息，指示客户端执行行动。
事件是从客户端发送到云的消息，通知Alexa有些事情已经发生。
-
步骤
授权
为了访问AVS API，产品需要获得一个LWA(a Login with Amazon)访问令牌，授权产品代表用户的意愿去调用API。有以下授权方式：
（1）远程授权（Remote Authorization ）远程授权用于设备，比如智能音箱。
配套应用程序授权 参考
https://blog.csdn.net/qq_27061049/article/details/80167376
关键：获取正确的Api_key。授权成功后获得access token。token需放在event的Header上发送给AVS
配套网页授权
（2）本地授权（Local Authorization） 本地授权用于Android和iOS应用程序。
（3）基于code（Code Based Linking）通常用于输入受限的产品，如电视或智能手表
构建Http/2连接
（1）基本概念：
Frame:帧，HTTP/2中基本的协议单元，每一帧服务于不同的目的，例如Head Frame和Data Frame组成了基本的请求和响应
Stream：在HTTP/2连接中，一个独立的，在客户端和服务器间交互的双向一系列连续的帧。
Downchannel: 在HTTP/2连接中创建的流用于从云端向客户端分发指令，主要用于发送从云端首发的指令或是音频给客户端。
Cloud-initiated Directives：云端首发的指令，如用户通过App调节设备的音量，一个指令被发送给产品而没有相关的语音请求。
（2）维持HTTP/2连接需要做到：a)建立下行通道流（downchannel stream）b)同步产品与AVS组件状态
（3）建立下行通道流。打开AVS连接10s之内，客户端发送GET请求。请求如下
:method = GET
:scheme = https
:path = /{{API version}}/directives
authorization = Bearer {{YOUR_ACCESS_TOKEN}} 在Android中通过OkHTTP进行访问，代码如下
mClient = new OkHttpClient.Builder().connectTimeout(5, TimeUnit.SECONDS).writeTimeout(10, TimeUnit.SECONDS)
.readTimeout(10, TimeUnit.SECONDS).build();//构建请求
Request request = new Request.Builder().addHeader(AUTHORIZATION, BEARER + mAuthToken).addHeader
(CONTENT_TYPE, MULTI_DATA).url(AVS_DIRECTIVES_URL).build();
Log.i(TAG, "[http call] downchannel request");
mDownChannelCall = mClient.newCall(request);
Response downChannelRsp = mDownChannelCall.execute();
boolean isDownChannelOpen = downChannelRsp.isSuccessful();
Log.i(TAG, "[down channel]: " + downChannelRsp.code());
内部相应变量如下：
private final String AVS_DIRECTIVES_URL = "https://avs-alexa-na.amazon.com/v20160207/directives";
private final String AVS_EVENTS_URL = "https://avs-alexa-na.amazon.com/v20160207/events";
private final String AVS_EVENTS_PING = "https://avs-alexa-na.amazon.com/ping";
private final String BOUNDARY = "this-is-a-boundary";
private final String AUTHORIZATION = "authorization";
private final String BEARER = "Bearer ";
private final String CONTENT_TYPE = "content-type";
private final String MULTI_DATA = "multipart/form-data; boundary=" + BOUNDARY;
private final String DISPOSITION = "Content-Disposition";
private final String META_DATA = "form-data; name=\"metadata\"";
private final String AUDIO_DATA = "form-data; name=\"audio\"";
（4）同步组件状态
在现有的连接上在新的事件流上建立POST请求。该事件流可以在客户端接收到响应后关闭。下面是一个同步状态事件的例子
:method = POST
:scheme = https
:path = /{{API version}}/events
authorization = Bearer {{YOUR_ACCESS_TOKEN}}
content-type = multipart/form-data; boundary={{BOUNDARY_TERM_HERE}}
--{{BOUNDARY_TERM_HERE}}
Content-Disposition: form-data; name="metadata"
Content-Type: application/json; charset=UTF-8
{
"context": [
// This is an array of context objects that are used to communicate the
// state of all client components to Alexa. See Context for details.
],
"event": {
"header": {
"namespace": "System",
"name": "SynchronizeState",
"messageId": "{{STRING}}"
},
"payload": {
}
}
}
--{{BOUNDARY_TERM_HERE}}--Android代码
RequestBody requestBody = RequestBody.create(JSON_TYPE, JsonHelper.getSyncStateJson(mPlayToken, mMsgId)); RequestBody multiBody = new MultipartBody.Builder().setType(MultipartBody.FORM).addPart(Headers.of (DISPOSITION, META_DATA), requestBody).build(); Request request2 = new Request.Builder().url(AVS_EVENTS_URL).post(multiBody).addHeader(AUTHORIZATION, BEARER + mAuthToken).addHeader(CONTENT_TYPE, MULTI_DATA).build(); Log.i(TAG, "[http call] Synchronized state request"); mStateCall = mClient.newCall(request2); Response response2 = mStateCall.execute();
同步成功后，客户端就能通过这个连接发送事件events，接收指令directives.
（5）注意事项：
a)采集的音频应该按如下方式编码：
16bit Linear PCM (LPCM16)
16kHz sample rate
Single channel
Little endian byte order
b)HTTP/2连接仅能同时支持10个流，包括events stream、downchannel、ping。因此需要确保事件流在接收到响应后被关闭
c)许多库都设有读取超时，当客户端长时间没有接收到数据就读取超时。因为downchannel stream需要AVS和客户端保持打开的状态，而且该流可能长时间不发送数据给客户端，所以需要设置读取超时为至少60分钟。
（6）Ping and Timeout
当连接处于空闲状态时，应该每5分钟发送一次Ping帧
Sample Request
:method = GET
:scheme = https
:path = /ping
authorization = Bearer {{YOUR_ACCESS_TOKEN}}语音识别（SpeechRecognizer Interface）
https://developer.amazon.com/zh/docs/alexa-voice-service/speechrecognizer.html#stopcapture
1、语音采集
Android中通过AudioRecord类进行音频采集。大致过程如下
private final int AUDIO_SAMPLE_RATE = 16000; //采样率16000Hz private final int AUDIO_CHANNEL_CONFIG = AudioFormat.CHANNEL_IN_MONO;//单声道 private final int AUDIO_FORMAT = AudioFormat.ENCODING_PCM_16BIT;//音频量化位数，可以设置每个样本的分辨率为16位或者8位，16位将占用更多的空间和处理能力,表示的音频也更加接近真实。 private final int AUDIO_BUFFER_SIZE = AudioRecord.getMinBufferSize(AUDIO_SAMPLE_RATE, AUDIO_CHANNEL_CONFIG, AUDIO_FORMAT); //1280//1、构造AudioRecorder mAudioRecorder = new AudioRecord(MediaRecorder.AudioSource.MIC, AUDIO_SAMPLE_RATE, AUDIO_CHANNEL_CONFIG, AUDIO_FORMAT, AUDIO_BUFFER_SIZE);//(音频采集来源（麦克风），音频采样率，声道，音频采样精度，音频数据存放的缓冲区大小) //2、开始采集 mAudioRecorder.startRecording(); mRecording = true; //3、将采集的数据写入文件，AVS识别 mAudioRecordingThread = new Thread(new Runnable() { @Override public void run() { writeAudioDataToFile(); } }); mAudioRecordingThread.start(); //4、停止采集 mHandler.postDelayed(mStopAudioRunnable, MAX_TIME);
在同Alexa交互初始化后，麦克风始终保持打开状态直到出现如下状况
- 接收到stopCapture指令
- 流被Alexa Service关闭
- 用户手动关闭麦克风
2、语音发送给AVS ——Recognize Event
Recognize Event用于把用户的语音发送给AVS，该event由两部分message构成，第一部分为JSON格式的object,第二部分为由产品麦克风采集的音频字节流。
所有采集到的发送给AVS的音频应该按如下方式编码：
- 16bit Linear PCM
- 16kHz sample rate
- Single channel
- Little endian byte order
Sample Message
{
"context": [
// This is an array of context objects that are used to communicate the
// state of all client components to Alexa. See Context for details.
],
"event": {
"header": {
"namespace": "SpeechRecognizer",
"name": "Recognize",
"messageId": "{{STRING}}",
"dialogRequestId": "{{STRING}}"
},
"payload": {
"profile": "{{STRING}}",
"format": "{{STRING}}",
"initiator": {
"type": "{{STRING}}",
"payload": {
"wakeWordIndices": {
"startIndexInSamples": {{LONG}},
"endIndexInSamples": {{LONG}}
},
"token": "{{STRING}}"
}
}
}
}
}Binary Audio Attachment
每个Recognize event都要求有音频字节流，下面是每个音频字节流的请求头headers
Content-Disposition: form-data; name="audio"
Content-Type: application/octet-stream
{{BINARY AUDIO ATTACHMENT}}
对应Android请求如下
RequestBody audioBody = new MultipartBody.Builder().setType(MultipartBody.FORM).addPart(Headers.of (DISPOSITION, META_DATA), RequestBody.create(JSON_TYPE, JsonHelper.getRecognizeJson(mPlayToken, mMsgId, mDialogReqId))).addPart(Headers.of(DISPOSITION, AUDIO_DATA), streamBody).build(); Request request3 = new Request.Builder().url(AVS_EVENTS_URL).post(audioBody).addHeader(AUTHORIZATION, BEARER + mAuthToken).addHeader(CONTENT_TYPE, MULTI_DATA).build(); Log.i(TAG, "[http call] recognize audio request"); mAudioCall = mClient.newCall(request3);其中streambody即为麦克风采集的音频数据流，为下面的mRequestBodymRequestBody = new RequestBody() { @Override public MediaType contentType() { return AVSMgr.OCTET_TYPE; }//将音频缓冲区AUDIO_BUFFER_SIZE内的音频数据写入BufferedSink @Override public void writeTo(BufferedSink bufferedSink) throws IOException { int readSize; byte buffer[] = new byte[AUDIO_BUFFER_SIZE]; while (mRecording) { readSize = mAudioRecorder.read(buffer, 0, AUDIO_BUFFER_SIZE); if (AudioRecord.ERROR_INVALID_OPERATION != readSize) { try { int sizeCount = readSize / AMZN_AUDIO_BYTE_SIZE; if (sizeCount > 0) { byte[][] bytes = new byte[sizeCount][AMZN_AUDIO_BYTE_SIZE]; for (int i = 0, start = 0, end = AMZN_AUDIO_BYTE_SIZE; i < sizeCount; i++, start += AMZN_AUDIO_BYTE_SIZE, end += AMZN_AUDIO_BYTE_SIZE) { bytes[i] = Arrays.copyOfRange(buffer, start, end); bufferedSink.write(bytes[i]); } } mTimeCounter++; mCounter++; if (mTimeCounter > 4 && mCounter > FIRST_RECORD_COUNTER) { long sum = 0; for (int i = 0; i < buffer.length; i++) { sum += Math.abs(buffer[i]); } double rawAmplitude = sum / (double) readSize; mAmplitude = Math.max(rawAmplitude, mAmplitude); Log.d(TAG, "rawAmp: " + mAmplitude); if (rawAmplitude < 33.3 && rawAmplitude > 25) { Log.d(TAG, "no sound dectected"); mNoSoundCounter++; } else if (rawAmplitude > 33.3) { Log.d(TAG, "sound dectected"); mNoSoundCounter = 0; } mTimeCounter = 0; mAmplitude = 0; } if (mNoSoundCounter > 3) { stopRecording(); } } catch (IOException e) { e.printStackTrace(); Log.e(TAG, "Error writing to audio file", e); } } } Log.i(TAG, "Finish speaking"); if (null != mListener) { mListener.onFinishRecording(); } else { Log.e(TAG, "RecordMgr: mListener == null."); } } };
3、AVS 返回指令
a)停止抓取指令StopCapture Directive
当AVS识别到用户意图或检测到用户语音结束时，该指令将指示客户端停止抓取用户语音。当接收到该指令时，客户端需要关闭麦克风，停止监听用户语音。
Sample Message
{
"directive": {
"header": {
"namespace": "SpeechRecognizer",
"name": "StopCapture",
"messageId": "{{STRING}}",
"dialogRequestId": "{{STRING}}"
},
"payload": {
}
}
}b) ExpectSpeech Directive
当Alexa需要额外的信息去完成用户的请求时，AVS将发送ExpectSpeech Directive。它指示客户端打开麦克风，采集语音流。如果麦克风未在指定的时间内打开，客户端必须发送ExpectSpeechTimedOut event给AVS。
在与Alexa的多轮交互中，设备将接收到至少一个ExpectSpeech Directive，指示客户端监听用户语音。如果当前 initiator object包含在ExpectSpeech directive的 payload中，则在返回给Alexa的 Recognize event 中， initiator object也包含在里面。如果payload内不包含 initiator ，则在Recognize even也不包含initiator。
Sample Message
{
"directive": {
"header": {
"namespace": "SpeechRecognizer",
"name": "ExpectSpeech",
"messageId": "{{STRING}}",
"dialogRequestId": "{{STRING}}"
},
"payload": {
"timeoutInMilliseconds": {{LONG}},
"initiator": {{STRING}}
}
}
}
c) ExpectSpeechTimedOut Event
如果麦克风未在指定的时间内打开，客户端必须发送ExpectSpeechTimedOut event给AVS。
Sample Message
{
"event": {
"header": {
"namespace": "SpeechRecognizer",
"name": "ExpectSpeechTimedOut",
"messageId": "{{STRING}}",
},
"payload": {
}
}
}
4、语音合成SpeechSynthesizer Interface
当用户询问问题或做出请求时，语音合成接口用于发送Alexa语音回复。如当用户发送一个天气询问”What's the weather in Seattle?”时，AVS将会返回一个带有二进制音频的Speak directive给客户端。
Speak Directive
当客户端需要一个来自于Alexa的语音响应的时候，AVS发送Speak Directive给客户端。该指令发送给客户端包括多个部分的信息，一部分是JSON格式的指令，另一部分是二进制音频数据。
Sample Message
{
"directive": {
"header": {
"namespace": "SpeechSynthesizer",
"name": "Speak",
"messageId": "{{STRING}}",
"dialogRequestId": "{{STRING}}"
},
"payload": {
"url": "{{STRING}}",
"format": "{{STRING}}",
"token": "{{STRING}}"
}
}
}Binary Audio Attachment 下面是二进制音频文件的头信息
Content-Type: application/octet-stream
Content-ID: {{Audio Item CID}}
{{BINARY AUDIO ATTACHMENT}}SpeechStarted Event
当客户端处理完Speak Directive并且开始播放合成语音的时候，SpeechStarted Event应当被发送给AVS。
Sample Message
{
"event": {
"header": {
"namespace": "SpeechSynthesizer",
"name": "SpeechStarted",
"messageId": "{{STRING}}"
},
"payload": {
"token": "{{STRING}}"
}
}
}
SpeechFinished Event
当客户端处理完Speak Directive并且 Alexa TTS完全渲染给用户（语音播放完成），SpeechFinished Event应当被发送给AVS。如果播放没有结束，如用户通过"Alexa, stop"打断了播放的时候，SpeechFinished就不会被发送。
Sample Message
{
"event": {
"header": {
"namespace": "SpeechSynthesizer",
"name": "SpeechFinished",
"messageId": "{{STRING}}"
},
"payload": {
"token": "{{STRING}}"
} }
}