|
Pipecat C++ Client SDK 1.3.0
Connect native apps to Pipecat bots
|
The SmallWebRTC transport, SmallWebRTCTransport, connects the client to Pipecat bots over WebRTC, peer to peer, with Pipecat's SmallWebRTC transport. To build it, see its README.
Give a SmallWebRTCTransport to the client when you create it:
Then use the client as usual. Everything else, like callbacks and threads, works the same with every transport.
The transport sends a WebRTC offer to the bot's server, which answers with its own, and then connects to the bot directly.
Start endpoints, like Pipecat's development runner (python bot.py -t webrtc) and Pipecat Cloud, start a session when you ask for the webrtc transport, and answer with its ID. The transport then sends the offer to the same server, with the same headers:
The offer goes to the start endpoint's URL with /start at the end of its path replaced by /sessions/<sessionId>/api/offer.
If your app already knows where to send the offer, e.g. because your backend started the bot, connect directly with webrtcRequestParams. It has the offer endpoint's URL in endpoint, and optionally HTTP headers in headers and data for the bot in requestData:
Bots on other networks need STUN or TURN servers to be reached. They go in iceConfig, which start endpoints like Pipecat's development runner return too. Pipecat's development runner returns the servers it was started with (--ice-servers), or a public STUN server when you add {"enableDefaultIceServers", true} to the request:
Every key can also be in snake case, e.g. session_id or ice_config.
The transport sends the user's audio to the bot and receives the bot's, as 16-bit PCM. Choose the sample rate and channels with SmallWebRTCTransportOptions. They default to 16 kHz mono, and any sample rate works:
The user's audio is sent to the bot with Opus, in mono, which is what the bot listens to. The bot's audio is converted to the format of the options.
Send and read audio with the client's send_user_audio() and read_bot_audio(), from your audio threads. read_bot_audio() waits until it's time to play the bot's audio. Disconnecting wakes it up, so disconnect before stopping a thread that reads bot audio.
The bot's audio plays at the pace the bot sent it, 80 ms behind, so the network's ups and downs don't reach your app: audio that arrives late or out of order still plays in its turn, and lost audio is concealed. If audio keeps arriving late, it plays a little further behind, up to 300 ms, and catches up again during silence.
Send the user's audio continuously, as it's captured, and send silence while the user is muted: the bot needs it to tell when the user stops speaking. Until you send some, the transport tells the bot the user's audio is off.
The user's audio is sent as is, without echo cancellation. If your app plays the bot through speakers, use your platform's echo cancellation or headphones, or the bot will hear itself.
Each client needs its own SmallWebRTCTransport, and you can have several clients at the same time, e.g. to talk to several bots. A client can also disconnect and connect again as many times as you need.