Clicking on the “Run App” button in RStudio will launch your Shiny application. It’s like magic! But ever wondered what goes on behind the code launching your application? In this series, you will dive into the Shiny source code to better understand what happens. The first part was all about the UI, or HTML, side of your Shiny application. This second part is all about the server!
Table of Contents
Understanding Shiny, httpuv, and WebSocket Integration
If you’re developing a Shiny application, you’re using a package called httpuv—even if you weren’t aware of it. But what exactly is httpuv, and why is it crucial to the Shiny server part? To grasp this, we first need to understand how web applications work, focusing on two key concepts: HTTP requests and WebSockets.
HTTP Requests vs. WebSockets
Imagine you’re a student (a web browser) trying to get information from your teacher (a server). You write a letter 📝 (HTTP request) to your teacher, ask a question, and wait for a reply. Your teacher receives the letter, processes your question, and sends back a response (HTTP response) with the information. This process is how HTTP requests work, and they are unidirectional—you ask, and then you receive.
Now, let’s consider a live online tutoring session. Instead of sending letters back and forth, you and your teacher are engaged in a real-time chat (WebSocket) 💬. You can instantly send messages and get immediate responses. This dynamic, ongoing conversation represents how WebSockets work—they are bidirectional, meaning both the client and server can communicate with each other freely and without delay.
But why do we still need HTTP requests if we have WebSockets?
The answer is simple: HTTP requests are necessary to establish the initial connection. When you first access a web application, your browser sends an HTTP request to the server. This request fetches the HTML, CSS, and JavaScript files required to render the web page. These files also contain crucial information on what else needs to be done besides displaying some HTML. Once the page is loaded, WebSockets can be used to enable real-time communication for any interactive elements.
The Role of httpuv in Shiny
A Shiny app needs to be able to process and answer HTTP requests, and it needs to open a server WebSocket connection. httpuv is responsible for both of these tasks.
httpuv is a low-level library in R that provides support for handling both HTTP and WebSocket requests. This dual capability is essential for a Shiny application, which requires both to function correctly. Shiny relies on httpuv to manage these connections, allowing your app to serve web pages and provide real-time updates.
Diving into the Shiny Source Code: HTTP and WebSocket Handlers in Shiny
Let’s look at how this all works in the Shiny source code, starting with the server-side code.
Server-Side: HTTP and WebSocket Handlers
When you call runApp() in Shiny, it eventually calls the startApp() function, which is responsible for creating and starting the Shiny application on a specified port and host. Here’s how it works:
Creating HTTP and WebSocket Handlers:
- The function
createAppHandlers()in server.R generates handlers for both HTTP and WebSocket requests. These handlers are responsible for managing the behaviour of the Shiny app when it receives these requests. - For HTTP requests, the handler processes the request and returns the appropriate response, such as a web page or resource file (HTML, CSS, JavaScript, or others).
- For WebSocket connections, the handler manages the real-time communication between the client and server.
Attaching Handlers:
- Once the handlers are created, they are attached to the application through the
handlerManager. Both the HTTP and WebSocket handlers are attached to the root path (“/”). ThehandleManageruses theaddHandlerandaddWSHandlermethods for this.
Starting the HTTP/WebSocket Server:
- The
startServer()function from httpuv is called to create and start an HTTP/WebSocket server. This server listens on the specified port and handles incoming requests using the previously attached handlers.
Here’s a simplified breakdown of what this looks like in code:
appHandlers <- createAppHandlers(appObj$httpHandler, appObj$serverFuncSource)
handlerManager$addHandler(appHandlers$http, "/", tail = TRUE)
handlerManager$addWSHandler(appHandlers$ws, "/", tail = TRUE)
httpuvApp <- handlerManager$createHttpuvApp()
return(startServer(host, port, httpuvApp))
Obviously more stuff is happening, including all sort of checks, but this is just for the sake of simplicity.
Back to our WebSocket! As you know now, the WebSocket handler (appHandlers$ws) is crucial for enabling real-time communication. It ensures that each client gets a dedicated WebSocket connection, isolating user sessions and preventing cross-communication between different clients. Now let’s zoom in a in a bit on this WebSocket handler and let’s break down how this handler works in simple terms.
appHandlers$ws contains a function (ws = function(ws)). This function gets called whenever a new WebSocket connection is made to the server. The main tasks it performs include:
- Security Check: It ensures the incoming connection is authorised.
- Handling Special Requests: It checks if the connection is for a specific purpose, like auto-reloading in development mode (which is really cool by the way!).
- Session Management: It creates and manages a Shiny session for each connection.
- Message Handling: It listens for messages from the client and responds to them.
- Cleanup: It cleans up resources when the connection is closed.
And of course there’s code involved with each of these main tasks:
Security Check
if (!checkSharedSecret(ws$request$HTTP_SHINY_SHARED_SECRET)) {
ws$close()
return(TRUE)
}
The handler first checks if the WebSocket request contains a valid shared secret (HTTP_SHINY_SHARED_SECRET). This is a security measure to ensure that only authorised connections are allowed. If the secret is invalid, the connection is closed immediately.
Auto-Reload Handling
if (identical(ws$request$PATH_INFO, "/autoreload/")) {
if (!get_devmode_option("shiny.autoreload", FALSE)) {
ws$close()
return(TRUE)
}
callbackHandle <- autoReloadCallbacks$register(function() {
ws$send("autoreload")
ws$close()
})
ws$onClose(function() {
callbackHandle()
})
return(TRUE)
}
If the WebSocket request is for auto-reloading (/autoreload/), the handler checks if auto-reloading is enabled in development mode. If it is, it registers a callback to reload the application when necessary. The connection is closed after sending the “autoreload” signal.
Session Management
shinysession <- ShinySession$new(ws)
appsByToken$set(shinysession$token, shinysession)
shinysession$setShowcase(.globals$showcaseDefault)
A new Shiny session is created for the WebSocket connection. This session will manage all interactions between the client and server. The session is registered using a unique token, and showcase mode (if enabled) is set for the session.
Message Handling
messageHandler <- function(binary, msg) {
withReactiveDomain(shinysession, {
...
shinysession$dispatch(msg)
...
})
}
A messageHandler function is defined to process incoming messages from the client. This function decodes the message, processes it according to its type (e.g., init, update), and dispatches it to the right handler within the session.
ws$onMessage(function(binary, msg) {
withLogErrors(messageHandler(binary, msg))
})
The ws$onMessage function is used to attach the messageHandler to the WebSocket connection, ensuring that it is called whenever a new message arrives.
Connection Cleanup
ws$onClose(function() {
shinysession$wsClosed()
appsByToken$remove(shinysession$token)
appsNeedingFlush$remove(shinysession$token)
})
When the WebSocket connection is closed, this cleanup function is triggered. It notifies the session that the connection is closed, removes the session from the active sessions list, and cleans up any resources related to the session.
A lot of technical talk, but getting a sense of how the WebSocket is handled on the R side shines some light on how your Shiny app actually works.
Client-Side: JavaScript and WebSocket Management
Besides the R side, there’s also the client side. On the client side, JavaScript (or, in the case of Shiny, TypeScript) plays an important role in managing the WebSocket connection. Let’s explore how this works.
WebSocket Creation in JavaScript
When your Shiny application loads in a browser, a WebSocket connection is established for real-time communication. This is managed by JavaScript in the shinyapp.ts file. Here’s how the WebSocket is created:
createSocket(): ShinyWebSocket {
const createSocketFunc: () => ShinyWebSocket =
getShinyCreateWebsocket() ||
(() => {
let protocol = "ws:";
if (window.location.protocol === "https:") protocol = "wss:";
let defaultPath = window.location.pathname;
if (!/^([$#!&-;=?-[\]_a-z~]|%[0-9a-fA-F]{2})+$/.test(defaultPath)) {
defaultPath = encodeURI(defaultPath);
if (isQt()) {
defaultPath = encodeURI(defaultPath);
}
}
if (!/\/$/.test(defaultPath)) defaultPath += "/";
defaultPath += "websocket/";
const ws: ShinyWebSocket = new WebSocket(
protocol + "//" + window.location.host + defaultPath
);
ws.binaryType = "arraybuffer";
return ws;
});
const socket = createSocketFunc();
let hasOpened = false;
socket.onopen = () => {
hasOpened = true;
$(document).trigger({
type: "shiny:connected",
socket: socket,
});
this.onConnected();
socket.send(
JSON.stringify({
method: "init",
data: this.$initialInput,
})
);
while (this.$pendingMessages.length) {
const msg = this.$pendingMessages.shift();
socket.send(msg as string);
}
this.startActionQueueLoop();
};
socket.onmessage = (e) => {
this.taskQueue.enqueue(async () => await this.dispatchMessage(e.data));
};
socket.onclose = (e) => {
const restarting = e.code === 1012;
if (hasOpened) {
$(document).trigger({
type: "shiny:disconnected",
socket: socket,
});
this.$notifyDisconnected();
}
this.onDisconnected(restarting);
this.$removeSocket();
};
return socket;
}
Breaking Down the TypeScript Code
Daunting? Perhaps! But let’s try to break it down a little bit.
WebSocket Protocol Selection:
Depending on whether the page is served over HTTP or HTTPS, the WebSocket protocol (
ws:orwss:) is selected accordingly.WebSocket Initialization:
The WebSocket is initialized with the correct URL, which is constructed based on the current window location and protocol.
Handling WebSocket Events:
onopen: Triggered when the WebSocket connection is successfully opened. The browser sends an initial message (init) containing any data the server needs to know about the client’s state.onmessage: Triggered whenever a message is received from the server. The received data is processed and handled by the application.onclose: Triggered when the WebSocket connection is closed. This could happen due to various reasons, such as server-side disconnection or network issues.Event-Driven Architecture:
The WebSocket events (
onopen,onmessage, andonclose) are linked to specific functions in the Shiny application, enabling real-time updates and user interaction.
Connecting R to TypeScript
So we have some R code, and some JavaScript (or TypeScript) code. How do they work together?
The WebSocket enables real-time, two-way communication between the server (R) and the client (JavaScript). This is where the ws handler from the code comes into play.
When a user interacts with the Shiny app (e.g., by clicking a button), a message is sent from the browser to the server via the WebSocket. The server processes the input in R and sends back updates to the client via the same WebSocket connection.
Since R and JavaScript are different languages, there’s a handler needed. The WebSocket handler processes incoming messages from the client, which are typically serialised JavaScript objects or events, and converts them into R data structures. Similarly, R outputs or instructions are serialized and sent back as messages that the browser’s JavaScript can interpret.
Now they can understand each other!
The Big Picture
To summarise, this happens to the server-side of your Shiny application when you click on the magic “Run App” button:
- HTTP requests are used initially to load the web page by fetching HTML, CSS, and JavaScript files from the server.
- WebSockets take over once the page is loaded, allowing for real-time, bidirectional communication between the client and server.
- httpuv is the backbone that enables both HTTP and WebSocket communication in Shiny applications, while JavaScript on the client side manages the WebSocket connections and handles events.
Resources
Some good stuff has been written on how Shiny and WebSockets in general work! Here are some resources that I used for this article:
- John Coene’s book “JavaScript for R”, specifically chapter 11.
- “Outstanding Interfaces with Shiny” by David Granjon, and the chapter about WebSockets.
- WebSockets 101
Wrap up
Hopefully this article uncovered some magic in the Shiny source code!
☕️ Was this useful to you and would you like to support me? You can buy me a coffee!
📚 Want to learn more about Shiny? Discover my online Shiny courses: ProductioniZing Shiny & CustomiZing WidgetZ.
I provide R and Shiny consultancy. Do you need help with a project? Reach out and let’s have a chat!