James Blaire & Barret Schloerke | Integrating R With Plumber APIs | RStudio (2020)
James Blaire & Barret Schloerke | Integrating R With Plumber APIs | RStudio (2020)
James Blaire & Barret Schloerke | Integrating R With Plumber APIs | RStudio (2020)
Thank you, everyonewho joined in today. I really appreciateyour time and hope that this can be aproductive time for all of us as we spend some time talkingabout R and Plumber together. As Robert mentioned,my name is James Blair. I work as a solutionsengineer for RStudio. And we’re joinedon the line today by Barret Schloerke who alsoworks here at RStudio and is the primary maintainerof the Plumber package that is here tohelp with questions and provide some supportthat way as well. So we’re gratefulfor his presence as well as we get started today. Our topic today isexpanding our horizons integrating our PlumberAPIs the idea here is I am hopeful that this reallyis an opportunity for if you’ve never been exposedto Plumber before.I hope you findsomething that’s useful. And if you’ve usedPlumber in the past and have come to seewhat’s new with Plumber. I also hope that there’ssomething useful. So the ideal state here isthat everyone walks away with something that’sbeneficial for them and that they feel likethey’ve learned just kind of give you an idea of whatwe’re going to do today. First, I want to setthe stage with a problem that we’re going to look at twotry to frame the conversation that we have. And then we’ll have a briefdiscussion about what an API is. In case you’reunfamiliar and then we’ll talk about how you getstarted using APIs with R and the Plumber package.And then we’ll spendsome time discussing some of the new features thathave been recently released in the latest CRANversion of Plumber. And then finally, we’ll concludeby looking at different ways that you can deploy these APIs. So they can be widelyutilized either inside of your organizationor by a broader audience. And then at the end,I’ll provide some links to additional resourcesand like Robert mentioned, this is being recorded. Today’s October 28 2020 so ifyou’re watching this recording down the road, things may havechanged in the Plumber package. And I’ll show somethingto you at the end that can make sure you stayup to date with the latest changes. And things like that. But resources willbe provided if you want to learn more today, we’regoing to work with a data set. This is the PalmerPenguin’s data set. If you’re unfamiliar this is afairly kind of recent to the R space data set, you canfind some more information at Alison Horsts GitHubrepository that I have linked here in the slides.But basically, theidea here is we have 344 differentobservations for three different penguinspecies: Chinstrap, Gentoo and this Adelie. And if you’re familiarwith the iris data set that the typical kindof classification data set you often see this isa similar style data set. Right fairly easy tounderstand down here on the bottom righthand corner, you can see kind of a previewwhat the data looks like. We’ve got the speciesthat island they came from. Do we have somedifferent measurements from each of thesedifferent penguins. Their bill length, billdepth, flipper length, body mass And then we have the sex ofthe penguin and then the year that the measurement took place. So what we wantto do here is just kind of frame this and thiscould be any data set right. This is the example thatwe walk through today is not specific to thisdata set by any stretch of the imagination. But I think likemost data science most analytical problems orprojects start with some data and that felt like a naturalplace to start today.Let’s start with somedata and go from there. So what we want todo now is we will say we’ve been handed this data,and we’re going to go ahead and we’re going to do thedata science or in some cases, you know means buildthe model, and we’re going to take thesethree or 44 observations. This is not a model building ornecessarily data science themed webinar. So this is not about how tobuild the best model possible. This is just let’s work with anexample that that’s easy for us to understand. So we’ll take the data you knowwe might explore it clean it up a little bit. And then we’ll fitthis model to it. And the idea here is givensome set of measurements. So given the billlength, the bill depth, the flipper lengthand the body mass, we want to predict the speciesof that particular penguin.Right and so that’s whatwe’ve done in the example that we have pulledup on screen. We fit this model usingthe tidymodels framework and then we’re able to seesome output from that model. And now that we have this, wecan generate new predictions right. We can given a setof new data that we get a new set ofmeasurements from a penguin of unknown species. And we want to predict whatthe species of penguin is. We can pass those newvalues through to the model and generate a predictedoutcome in terms of what species we think is thatpenguin is likely to be. Now this is all great,but the question now is kind of, well, now what, right? We we have a couple of options. Now now that we’vedone our work. We’ve built this model, wehave a few options ahead of us. One is we can justkind of ad hoc score new penguin predictionswhenever we receive new data.Right So maybe once a month. We get an email from researchersthat are off researching penguins and they sendus their new set of data and we run that sort ofdata through the model in order to predict the speciesof these new measurements. That works right. But maybe we wantsomething that’s a little bit more real time. Maybe there’s maybemeasurements are happening at a pretty rapid rate. And we don’t want ourentire day job from here on out to be predictingpenguin species using the model that we built right. But we also don’twant the model that we built to just kind ofwither away and die. We spend a lot of time here. This was something thatwe feel like contributes to the goals of our businessor the goals of the project. We’re working on. And we want to make sure thatit continues to contribute in the best way possible. But the problem is we’re kindof stuck because we either have to be theones that generate new values and newpredictions because we’re the ones that built the model.And we know how touse R or we have to find somebody elsethat knows are and can do that themselves right. So we maybe we emailthem or we provide them with access to themodel and they’re able to generatetheir own predictions. But that’s stillkind of a high bar because that meansthat that person needs to be able to open up an Rsession and load our model and load the data and run thedata through the model, which if you’re an Ruser that’s probably not that big of a deal.But if you’re trying to teachone of the business users or one of end users ofyour model how to run this and you’ve got to teach themabout are now all the sudden, this becomes a much larger task. The other option is wecould just pass this off to somebody else we couldsay, OK, we’ve done our job. Here’s the model. Here’s kind of howwe set this up. Now let’s let that modelgrow up quote unquote. And we’ll give it toan engineering team and they’ll reconstructthis model in a framework that they’re using. So that it can be used forreal time scoring or real time predicting of new data and thatthat’s a totally valid option, but it’s often a little bittricky to get that right.Right to be able to handoff the model in that way, it’s time consuming. It it can be adifficult process to go through when inreality, we already have everything we need. What we really need is just aconvenient way for other people to be able to use our model. And that’s reallywhere APIs come in. So let’s take a step asidefrom the model for a moment. And let’s just talk aboutthis notion of what an APIs and in general, the API standsfor Application Programming Interface. Now that can mean a lotof different things. And in different contexts,the idea of an API can mean different things,which makes it a little bit difficult to understand. And so for the purposeof our conversation today we’re going to narrowin on this definition that we can get an APIs a webAPI communicating over a HTTP.And basically what thatis that provides us with a standardizedway for different can for different computers tocommunicate with one another. I get answer that. And I’m not going to diveinto the nitty gritty details of what makes up an API requestand what a response looks like. There’s lots of differentfeatures involved here that you can lookinto and that and that will provide some resourcesthat provide some detail there. But the main ideais when we talk about web APIs overHTTP, which is what we’re going to talk about today. This is just a standardizedset way for different computers to communicate. Just like I’m communicating. Now through my voice. And then you’re ableto understand computers have their own way ofcommunicating HTTP web APIs one of those mechanismsfor communication. The idea is there’sa client that sends a request that requestmeets the standard of what the server is expecting. And then a serverreceives that request and then generates somesort of a response. And that response canbe something as simple as, I received the request.That’s all that’s all I need. That’s all I need todo to something that’s much more complex like firingoff some sort of process or updating adatabase or generating predictions against a model orany number of different things can happen in response toa request that comes in, but the server is responsiblefor taking the request and doing something with it. And then providing a response. And even if you’ve never reallyinteracted directly knowingly with API in the past. And it’s interestingto note that anytime you use a lot of web browsers. If you use Google Chromeor Safari or Brave or Firefox or anything like that. And you visit a website. There’s a requestthat your browser makes to a serverthat says, hey, I’d like to view this web page. That’s that’s at this address. And the server responds withtypically a bunch of html that your browser, thenrenders for you to view.So anytime you viewa website, you’re really interacting with API. Even if you don’tacknowledge that fact right. Your computer is a clientand it’s sending a request. And there’s a serversomewhere that responding to that request with some emailthat you view it as a web page. Now this is, again,not meant to be a comprehensive overview ofwhat APIs are or how APIs work. But this hopefullysets the stage and gives you a littlebit of background and the next general question. I think that makessense to ask here is.OK, well, that’s great. But as an R useras a statistician as an actuary as a datascientist as an analyst why do I care right. Why is this somethingthat’s important to me. And I think there’skind of two reasons here that thisbecomes significant. And it goes back toour initial conundrum where we built this model. And we want to givepeople access to it. But we don’t want totrain the whole company on how to use our right. And the fact isAPIs allow the work that you do to be used by a widerange of tools and technologies at that. And that that’s nolonger no longer limited to just the language. So I can nowessentially say, OK, I’ve got this model thatwe’re going to work with them will walk through an example. This is the momentI this model here. And I’ve got another team inmy organization that’s using Python or they’re using Cor they’re using JavaScript or they’re using C+ or they’reusing any number of tools and they want to use my model.Well APIs allow me to do thatin a very, very straightforward and easy way. So that I no longerneed to hand off my model to be rebuiltby another team in another language. But instead they canjust communicate directly with the work that I’vedone via this web API. And this again, the ideahere is this dramatically reduces the handoffbetween the work that you do an Rand other tools that are being built withinyour organization by teams using other languages,or other technology frameworks.So with that kind of in mind,let’s talk briefly about how Plumber works andhow Plumber creates this bridge betweenour user and developers and other tools andtechnologies here’s an example of a Plumber apiover here on the left hand side of the screen. And I want to I’m goingto walk through some of the core components here. But before I do that,if you just take a step back and ignore the comments. Right, so if you just ignorethe comment that lines of code or the commentedlines in this are code it should look prettystraightforward. If you’ve writtenR code before. We’ve got a couple offunctions that we define and we load the Plumber packageat the top of the script. And if I look atthese functions. There’s nothing that’s terriblyinteresting or different about them. If I look at the firstfunction over here it takes a singleparameter message.And then it returns alist just that verifies that it received the messages. Essentially right. It returns a list thatsays, OK, the messages and then it echoesback that message. My second function here. It doesn’t take any arguments. And instead it just returns ahistogram of some random values that I draw from thenormal distribution. So it’s pretty easyto reason about what these two functions will do.Right? Function oneis going to echo back whatever I giveit function two is going to give me a randomhistogram of some values. Now the real…. I think the real magic andthe real power of the Plumber package is how it allowsme to go from these simple R functions toa responsive web API in just acouple of comments. And that’s where thesecomments come into play. So if you focus on what I havehighlighted here in this box, you’ll notice thatthese comments. Look a little bit different. The standard are comments. I’ve got the poundsomewhere the hashtag symbol that’s what is typicallyused to comment things in R, but then I followthat with an asterisk. And that indicatesthe Plumber hey, this is a special commentthat provides instructions for what Plumber needs to do. And so I can see, for example,if I take this first comment I give it this APItitle tag and I know that because it startswith at symbol and an API title and then I give the APIsome sort of a name Plumber example, a pan.This case. And then the next comment blockthese three comments following describe how Plumbershould use the function that comes afterwards. So a first comment here tellsme, gives a description of what this function is going to do. It’s going to echoback the input. My second comment heredescribes the parameters of that function. So in this case have asingle message parameter that parameter is the messagethat I’m going to echo. And then the finalline here is perhaps the most important becauseit defines both what types of requests. This will respond toas well as what path. This particular function islistening on and so essentially what this says is OK ifI run this as an API. And I make some sort of arequest to the Echo path you see this /echo here, thenwhat I’m going to get back is I’m going to get back theresponse of this function and whatever thisfunction does that will be returned towhatever client makes a request at this echo endpoint. And to be even morespecific that client will make a Git request,which we’ve denoted with this get tag here.So the idea is a rightstandard our code as functions. I use these special commentsto identify how Plumbers should handle those functionsand then I run the API or I plumb it, right? And you can see here inthis little screenshot I’ve had this run API button. And in practice, whatthis looks like is this. I click Run API, I’ll getthis nice little window that pops open over here thatshows me the API running, and then I can try this out.I can say, OK, here’s mymessage is Hello, world. I’m going to go aheadand try that out. And if we scrolldown, we can see that the outputof that message is the out or the outputof that execution is the output of the functionthat I wrote in R. So again, if I lookat this try this out write my message Hello, world. And then here atthe bottom we’ll see some JSON thatcomes back that contains the output of myfunction message the messages Hello, world.And it really is that simple. Right now what I’ve done isI’ve taken my functions in R and I’ve made them. So that they’re easilyaccessible from other tools and other frameworks. So let’s actuallytake a look at this. And let’s consider our originalkind of situation here. We’ve got this model that webuilt using this penguin data set and what we want todo now is we want to see, OK, is there a way forus to expose this model. So that others caninteract with it. Is there a way forus to essentially say, OK, what if a clientmakes a request with new data. Could we just respond withthe predicted outcomes.Could we just respondwith the predicted species given that request anddo that automatically. So that if somebodyin another department has a Python script thatwants to use our model they can just send arequest with new data. And we can send them backthe predicted outcome. That’s what we wantto accomplish here. So what’s actually going toflip over here to RStudio. Now and let’s go aheadand let’s build this out. OK So what we want to do is wewant to load the final package. And then before Ido anything else, let’s just kind ofmake sure that we’ve got the pieces in place thatwe want to have in place.So what I’m goingto do first here is I’m going to create(And zoom in a little bit)list . I’m going to create,just as a single endpoint that all it doesis just verifies that things are working. It’s not going todo anything other than just return someinformation that says, hey, the lights are on. Things seem to be working. So we’ll call this, we’llwrite the function here. Like a type. And we’ll say we wantto return a list. And we want to say, OK,the status is that all. And then we’llreturn the time, just so we can verifythat this is working the way that it should be. And if I run this function, wecould call this function status or something like that.And if I run this function inR, we can see what I get back. I get back this list. And it tells me thestatus is all good. And it gives me this timestampof when that function executed and now what I want todo is I want to turn this into an API endpoint. So we’ll give it alittle description here we’ll say we’ll call this help. Let’s call this health checkright in the API running. So that’s our description. And then this willrespond to get requests at the help desk right. So I’ve taken I’vewritten my function. And now all I’m doing is I’madding these special comments that we saw beforethat tell Plumber what to do with this function. All right. What do I want to do. Well I want to callit health check.This is the titleand description. I’m going to give it. And then here’s theimportant piece it says, OK. Whenever I get requestcomes in and I’m not going to get intowhat different request types are and things like that. That’s that’s an exercisefor another time. But essentially, if a requestcomes into this location, then return the results of thisfunction execute this function and return the results. And if we save this. We can click. Run API here. And we’ll see over hereon the right hand side, we’ve got thislittle interface that shows up that allows us to see. OK, we’ve got this healthcheck let me expand this out a little bit. So we can see. All right. So here I’ve got this. Get health check it tells methe description of the endpoint, which is what we provided. Right here is the API runninghealth check is even running and we could try this out.Let’s come in and try this out. There’s no parameters. I don’t need to provideany parameters here. So I just execute this. And here I see my response body. And I can see directly. In fact, let’s see if we canmake this a little easier to compare. If I scroll up, up. Here we go. Here’s the result ofmy function in R right. We we ran the function inR and that’s what we see.And here is the result whenI make a request an API request to this endpoint. And hopefully you can see theconnection between the two. Right I hear youup, up top, I’ve got an R list that wascreated that tells me the status of the timeand hear down below. I have that same list except nowit’s just in JSON format, which is kind of the industry standardfor how these APIs communicate.Now you can adjust howthis what type of response. This an API generatesand we’ll look at that here in a little bit. But by default Plumber willtake the output of your function and convert it to JSON. And that JSON data willbe returned to the client. So here’s what we here’s whatwe saw return to the client. And basically what this says isOK, we have this working right. Is working. We’re able to seethat the APIs running. We can try this out again,if we wanted to write we could execute this again. And we see the sameresult. But the timestamp is now updated to reflectthe current time, which in my local time is 11 23. All right. We’re going to stop the API. And now we’re goingto keep going.So this makes sure that. OK, we’ve got allthe pieces in place. We know what we’re doing. Now let’s figure out, OK,what do we really want to do. What’s our goal here. Well, first of all,I need the model. So I’ve already saved the model. We’ll just read the model in. We’ll see a model is read model.OK So I’ve got this model onfile that I’ve already saved. We can read this in here. Right So we’ve our model. And now what I wantto be able to do is I want to say, OK, whatif I predict my model, and I say new data isand I’ve got some JSON data lying around. I’ll say read JSON Collins. I don’t think Clyde was OK. So let’s see Oh, we need to do. We go. All right. So now if I lookat this and I’ll walk through what we did here. But if I look whatI really want is I want to say, OK, given mymodel that I already trained I want to be able togive it some new data and return thepredicted outcomes. So in this case,what I’m returning is I’m returning it aprobability value for each of the three different species.Right So in this casepenguin 1 has a straw where we’re almost entirelyconfident as 2 prediction to our observationtwo or very strongly predicting that it’s adelie. So on and so forth. I can see the breakdownacross each potential outcome or each potential specieshere in my output. So this works. This is I mean, this is how Iwould generate new predictions in R, but what Ireally want is I really want this functionalityto take place in an API.I want somebody to be ableto just pass me some data. And then I can pass themback the predicted outcomes like what I’m doing here. So this is our goalright here our goal is to do this is to have thiskind of behavior in an API endpoint. So we’ve got our model in place. We’ll need to. Let’s bring in. We’ve got a coupleof other packages that we need to makesure that they’re available in our environmenthere just for the model. So we need to bringin the past Netlify package for the predict functionto work to the right, the way that we want it to.And this is all becauseof how we train the model. And then we want to bringin the Ranger package because we train this modelusing the Ranger package. So this is, again, justto remind ourselves. This is a random forest modelthat we built. We saved it out. And now we’re goingto use it in this API that we’re building OK, I’vegot these pieces in place. So now what I want is I want tosay, OK, I’ve got a function, and what I want to be ableto do is I want to say, OK, I want topredict I’ve already got my model because I’veloaded that in my environments.I’ve got my model andthen I’ve got new data. And this is the part. OK, now I’ve gotto figure this out because where is thisdata going to come from. Right I don’t have likeit like when I run. This is an API. I want this data tocome from the user. Right this is data that comesfrom the client request.So OK, let’s thinkabout how that works. The way that this worksand Plumber is I’m going to have I’m goingto write my function. So that it takes a coupleof arguments request our IQ and response are yes. And what this does, if Iwrite my function this way, it allows me to have access tothe full request that’s being made from within my function. Plumber will automaticallypass that request object into my function. And then I have access to it. And one of the cool thingswith Plumber with the latest release of Plumber isthat you can automatically pass incoming data.And make it availablein the request itself. So so for example,if I have a user that makes a request withsome JSON data and says, OK, here’s some here’ssome information about some newpenguins we found. And I want to knowwhat their species is. So they make this request. Plumber will automaticallytake that JSON data converted to a data frame. And make it availablefor me to use within the body of this functionthat I’m writing right now. So all we need to knowis we need to know. OK, well, what. Like where does that get stored. And it gets stored as therequest as part of the request. And it gets stored asthe body object attached to this incoming request. And in fact, we’ll takea look at the moment. But this is essentially whatI want to be able to do. I want to say,OK, let’s do this. And let’s say typeis probability like we saw before right. We’re just we’re just takingthis function that was our goal and we’re now puttingit inside this function that we’re going touse within our API.So I’m going tosay this is going to be predict speciesfor new penguins and then this islet’s say we want to respond to POST requestsat the predict end point. OK, excellent. Let’s let’s go and run this. I’m going to go. I’m going to go out andcomment this out just because this isn’t a necessarypiece of what we’re doing. This is just so thatwe know what we’re trying to accomplish here. We’re going to run this API. And let’s pop thisopen over here. OK So we see our health check. Let’s just verifythat that’s doing what we think it should be. And it looks like it is. Which is great. That means everything’sworking on the Plumber side as far as we know.And then let’s take a lookat predict and point here. OK, we’ve got thispredict end point. We’ll try it out. OK execute OK. We get some sort of error here. OK And if we look atthis, this is just an R error that’s comeback to us in JSON format. Right which is, again, a nicekind of feature of a Plumber is that if my arcothrows an error. I can capture that error, and Ican gracefully return something to user or if I don’t captureit or do anything like that, then Plumber willautomatically take that error message converted into JSONand return it back to the user. So they get some ideaof what’s going on and what’s reallyhappening here is I’d like I didn’tgive this any data. So this is saying,look, I didn’t find one or more independentvariables was not found like I didn’tfind any data, then that actually makes sense,because I didn’t I didn’t give it any data in here. Right like I didn’t. There wasn’t any part ofthis where I like provided it with some sample data.And so now I’ve got to figureout, OK, well, how do I how can I providesome sample data here. Right and one of the featuresthat’s new to Plumber is the ability tomodify the user interface that that appears. And so we saw thisuser interface that showed up over you in fact, letme just run this one more time. So we see it. Right So I’ve gotthis user interface that appears over here. This is really,really nice because it means that is an R userwhen I’m building my API. And I run it. I get a nice really cleanway to interact with the API directly from within RStudio. Right right here inRStudio I can check to see that my APIs working. And here we go. Right things are working. I can check this end point. But as we just saw if Itried to do something like. Check this endpointall of a sudden, I’m in a little bit of a bindbecause there’s nothing here that allows me to pass in data.So what I really would like isI’d like some way to say, OK, how can I like, can I justplop some JSON in here right. Instead of instead of notbeing able to do anything. What if I could justdrop some JSON into here. And run it that way. And with the newestrelease of Plumber that is entirely possible. So that entire user interfacethat we’re just looking at is built around something calledthe open API specification. Now the entire openAPI specification is a massive standardizationstandardized format for defining API behavior andendpoints and things like that. I am not going tospend a lot of time going into the entirety ofwhat open APIs and all of that entails. You’re certainly welcome tolook more into that on your own.But what it meansfor us as our users. And as Plumberusers is that if we want to enhance the capabilityof our user interface here we can do so bydoing one of two things. We can either modifyit directly in Ar with by modifying alist a massive list, or we can provide ourown specification file and use that from within our. So that’s what we’regoing to do here and I’ll show youwhat this looks like. So I’m going to createanother little section here. I’m going to callthis at Plumber. This is another newfeature of the package. And basically whatthis special tag does is it allows me to takemy existing Plumber object and modify it.However, I want. So I’m no longerdefining an end point, but rather what I’mdoing is I’m saying, OK after I’ve done all this stuff. So after I’ve added healthcheck after I’ve added predict. I want to further modify thisobject that’s being created. And so I’m going tosay this is a function and we’ll call it PRfor my Plumber router. And then in here,I can modify what I want to do to this PR object. And this gives meanother chance to unveil one more really great featureof the news release of Plumber. And that is I can now dosomething like this PR pipe. We are set API specand I want to say, OK, let’s get this GMO filethat I’m going to read. And I’ll show you what thisviolence like in just a moment here.OK So let’s before welook at this yamma file. Let’s look at whatwe’re doing here. I’m saying, OK Plumbertake this information and start buildingme an API router. But now that you’vebuilt these pieces. So you’ve got this endpoint you get the template. I want to dosomething else to it. I want to take thatrouter and I want to add this APIspecification file. Right and so whatI can do here is I can either provide a namedlist, which this will do or I can modify the existingspecification that Plumber is already building for me.There one works. I kind of like thisapproach because it allows me to lay out thewhole outline of what I want. And that can be helpful. But either one works. So if we look at thisopen API file here. This is a llama filethat just contains information about the API. And notice I define like thesummary for my health check. I defined thesummary for predict. And if I run this. Now we come back overhere to my Plumber file. And we run this, you’ll noticethat my descriptions have now changed. Right my healthcheck says determine if the APIs running andlistening as expected. And notice thathere my description says health check isthe API running well. Why are those twothings different. They’re differentbecause I overwrite in this GMO file whatthat description should be noticed in here. I say determine if theAPIs running and listening is expected and that is whatwe see listed in my interface.So I have effectivelyoverwritten some of what I already did byproviding this new definition file. But in addition to overwritingsome of that information. I’ve also described that I wantto be able to pass and JSON data to this predict end point. And so if I come overhere and open this up. It tells me here. Look, the requestbody should be Jason. And here’s an exampleof what that JSON data should look like. And if I try this out. I can come in here. And I edit this,I could say, well, what if we saidthat you won tender. What if we said 54 40 or 5040, whatever the case is right. I can come in and I can passmy own data into this end point now and try it out. Right we can execute this. And here, we see that we have,once again, an internal error. And this is because if wecome back over to our API if off for just a moment, we’repassing in only a single value.And it’s being passed as avector and set up a data frame. So if we just changethis one piece and say we want this to be a data frame. Now let’s run the API onemore time, try this out. Here’s our datathat we’re passing in let’s execute on our data. And here we see our responseis a JSON object that contains thepredicted probability for each associate species. So in this case, this particularpenguin is almost a toss up right. We’re not super sure which oneit is it may be a chin strap but it’s a fairly evensplit across all three potential outcomes here. And what’s goodabout this is now we have exactly what wewere describing originally. Right if I come back uphere for just a moment. And look at whatmy goal was right. I want to enable an externaluser to give me some data.And then I want tobe able to return to them theassociated likelihood that that particularset of data matches up with each of thethree species that are our potential outcomes. And that’s exactlywhat I’ve done here. Now To recap right just torevisit this idea one more time. We have specified our ownspecification file this yama file that identifies allthe details of how this user interface should be laid out. And that’s where I can. That’s that’s what enables meto have the ability to input the JSON data needed to generatea response from this end point. So all from withinRStudio itself. I’m able to build my APIand I’m able to come over here and verify thatthat APIs working the way that it needs to OK. That’s been a littlebit to swallow. I know that maybe we’vegotten into some detail that we didn’t necessarilyneed to or want to get into.But I think it’suseful to understand how some of these pieces work. Is it necessary to changethe EPA specification file. No is it somethingthat you have to do. No right. You can do this. You can operate justfine without it. Now granted thatmight mean that you need to find some other wayto make a request to your API to verify that you are predict. End point is workingthe way that you expect.But there are lotsof tools that work that will enable youto interact with APIs and a really nice kindof clean interface. Postman is one thatI use regularly. And there areseveral others that exist as well, that allow youto generate requests and see responses and things like that. So you need to makethese modifications. No but if you wantto, you certainly can. Let’s take a lookat one more thing here that I think is, again,maybe a little bit extra, but I think it’s useful toknow the default interface.I going to run thisAPI one more time. The default interfacehere is swagger and I see that uphere on the top. Let him in court. This is the swaggerinterface swagger. It has been aroundfor quite a while. Open API thespecification kind of grew out of some of theefforts that went into swagger and everything they got. But essentially nowwhat we have is swagger is we’ll interpretmy specification. And give me this interface. There are otheralternatives to swagger that exist today thatdo the same thing.They create aninterface for me to use. And now within Plumbergiven the latest release it’s entirely possible to useone of those other systems instead of using swagger itself. So for example,if I wanted to use that there’s a library ora viewer called rapid arc. So if I wanted to use rapid arc. I could come in here. There’s an R package. It’s not released on CRAN it’sonly on GitHub at this point, there’s an artifact.It’s called rapid arcthat I can install here. And I could load here. And then further down. I could say, let’s do PR setdocs and we’ll say docs is OK. We’ll save these changes. And now if I come backand run my API again, you’ll see that myinterface looks different. I have the samefunctionality right. I can open up this oneI can try this out. I can see the response. I can open up this one. Come in here, I cansee the example data. I can try this out. We see the response right. So I get my functionalityremains the same, but maybe I have a preferenceover which particular UI use when I’m experimenting with theend and working with my API.And you have theflexibility now and Plumber to be able to define those. One more thing that Iwant to look at here is while we’ve got this open. We have this built an examplehere that gives us a sink. The measurements for asingle penguin but I’ve also got this job there. Excuse me. This JSON data here thatcontains measurements for several penguins. And if we copy this inhere and try this out, we’ll see that our response. Now includes predictedoutput for all the penguins that provided input for right. So I now have thislong JSON object that contains all theinformation for the penguins that I provided. OK, let’s get onemore thing here. Let’s come back intomy Plumber file. And let’s say, OKinstead of Jason though, let’s say I just Ireally want I really want CSV data to come out of this. Right So we couldsay at serialize our CSV this tellsPlumber look when you return the output of thisfunction don’t return it.As Jason returnedit as a CSV file. And if I run this againor return this as CSV did. And if I run thisagain and try this. Let’s bring in all ofour observations again, you can see now insteadof this JSON object. I have comma separatedvalues for my response. Right So I can adjust whatspecifically is returned to the client from within here. We’re going to take a stepback from this for a moment. We’ll revisit our APIhere in just a minute. Let’s go ahead and stop this. We’ll save these changes here. And let’s come back, ofcourse, to our slides. OK So just to recapand kind of summarize what we looked at hereseveral new features have landed in thePlumber package with the latest release. The tidy interface. So the ability to build yourAPIs by piping from one command to the next is new. We read barely touched on that. But that’s somethingthat you can explore more if you’reinterested in the ability to automatically pass and passthe incoming request body. So that it’s available fordownstream execution is new, which we saw that when wesaid, OK, the request body contains the past values thatwe’re getting from the client.There’s support for newserialized hours and the way that you define serializedis greatly simplified. The open APIspecification, we look at can adjust that in acouple of different ways by providing your own fileor by modifying the existing list that are is working with. You can customize the UI. So instead of using swaggerwhich is the default. You can use rapidly likeyou could use Reebok as a handful of differentcustomer UIs you can use. And last but not leastplumbed the Plumber hex logos got a face lift. This was on the title slide. So you probablyalready come across it. But there’s the newPlumber logo that done that accompaniesthis update the top. OK So now we’ve got this API. It’s running. It’s doing what we want. We mission accomplished right. We’re able to take in some data. We’re able togenerate predictions we’re able to return thatdata back to the client.All is good, exceptfor one thing we are still thebottleneck right like this is like I’m justworking off my laptop right now. And so when I’mrunning this API. It’s just running locallyhere on my own machine. And that works fordevelopment purposes. And testing purposes. But what happens whenI now want to say, look Sarah and engineeringis ready to make a request to this API.Well I don’t want my laptop toalways be the one thing that’s servicing these requests. I need some way to deploy this. So that it’s an input in anenvironment where it’s always listening andavailable to clients that are trying to makethat request again, whether those clients or otherparts of my organization, or maybe I’vebuilt an API that’s just generallypublicly available and is being usedby who knows who. There are a coupleof options that exist for deploying PlumberAPIs are RStudio Connect is a professional productthat’s developed and built by RStudio that allows foreasy deployment of Plumber API.We’ll look at an example ofthat here in just a moment. You can deploy you can wrap upthese guys inside of a Docker container and then deploy theminto an environment that’s suitable for thattype of deployment. And then there’s some nicekind of helper functions that used to exist inthe Plumber package and have now been offloadedto a new package called Plumber deploy that allowsyou to easily deploy to DigitalOcean. And that package may evolve toinclude other deployment points as well. So you’ve got a hostof different options. I want to spend acouple of minutes just to talk about ask you toconnect as one potential way of deploying these Plumberguys after you connect is, like I mentioned, it’sa professional product that we create anddevelop here at RStudio. It allows for easy pushbutton deployment of things created an R and Python.It handles dependencymanagement. So all of your packagesand dependencies come along for the ride. It allows you to adjusthow your API scales how it responds toconcurrent requests how new processes are generated. Things of that nature. It integrates with Gitand GitHub a GitHub. So you can automaticallydeploy from repositories. You can specify who haspermission and authentication to access things likeAPI and dashboards and things like that. And then like I said,in addition to Plumber you can also publishR Markdown documents and reports China applicationHuber notebooks flask streamlined Python applicationsand other additional other additional pieces of content. And if you’d liketo learn more, you can visit our website ourconnect which details what RStudio Connect is andit provides a little bit of additional context. But what I’d liketo do now is I’d like to take a lookat RStudio Connect.So if I come here. This is RStudio Connect let’scome in to the landing page here. So here’s what I see when I comeinto RStudio Connect something like this. And let’s go through the processof publishing this API that we built. Now I’ve set up this APIthat we built in RStudio to BI Publisher will you get.So if I come inhere and say, OK, let’s import thisfrom a Git repository let’s pull in therepository name. So what is let’s see. OK So here’s therepository for that. I find it really quick. So get I’ve got thisrepository that I keep talks and things that I do in. And then this is the Plumberwebinar 2020 repository. Click next year. This will look to see whatbranches are available. I’ve only got a single branchavailable there right now. Click next. And then this will look fordeployable sub directories. In this case, we only have one. It’s looking for this manifestfile that I previously created.And then we’ll call thisformer pay to play content. And now this is deployingour content, we go. Let’s open this up andsee what it looks like. OK So here we haveour API running. I can open this upand say, OK, let’s go. Anybody can come in here andview this if they want to. Here’s my health check.Let’s verify thatthis is working. Looks like it is gives me backthe information that I expect. Let’s check out our other endpoint or predict end point. It looks like it’s doingwhat we wanted to do. There we go. We see our CSV output here. And all is well. And good. Right I’ve got everything. Everything is working theway that I expected to. And just to illustrate this. Right if I so one way toquickly and easily demonstrate how an external either toolor framework or language can make a request to this APIif I open this API up here.Here we go. Let’s take a look at this. So now I’ve opened this up. So that it’s just openkind of in its own isolated environment. And if I instead am goingto darks go to help check me this might be alittle small to see. And I apologize. But I’m just visiting the URLand then adding health check to the end of it. And basically what thismeans is my browser is now going to make a requestto the health check end. And if we do thismy browser shows me the JSON that’s coming backfrom that health check point. So without any sort of likefancy bells and whistles I’ve just now made a request tothis API from my web browser.And I’m now seeing the resultsof that request rendered here and in front of us. OK So what does thatwant to highlight here before we wrap things up. All is well. And good. Right my guys running. I can give otherdevelopers or other teams within my organization, Ican give them the information they need to make requests. This API, they can make thoserequests my API will respond back to them and all is good.But now let’s say,OK, I published this. It’s great. And then I get an email anda couple of days says, look, we started using aninteracting with your API. We notice that is giving usback comma separated values. Right and we’d reallyprefer to get Jason like we can pass the CSV data. But our platformsbuilt around Jason, could you provide us withJason as a response instead of these comma separated values. And we’ve alreadyseen how to do that.If we come backinto RStudio here. And take a look. Right if we can if wetake this serialized and say instead ofCSV let’s just do Jason and Jason isthe default right. So if I deletedthis whole thing. It would still be Jason. But we could save this change. He’s OK. We’re we’re going toreturn JSON data here. Let’s go and save that. And then let’s come in tosee if I can pull it open.Here we go. Let’s come in to get andmake Ken commit this change. We’ll say, OK, we’re going toupdate to these song serial serialization. Commit that, we’ll goahead and push this change to get and now let’sgo ahead and grab our big bunch of penguins. And if we come back overhere RStudio Connect will check every15 minutes or so by default to seewhat changes have been made to the repository. But I’m also just goingto manually kick it off and say, OK, let’s justcheck for any updates in the repository.Hey some changes were found. We’re going to redeploy this. OK We’re ready to go. And now if I come in andtry this and point out will change our exampleto include all this. We can see that my responseis now Jason instead of CSV and I didn’t haveto do anything. All I did was make a changeand commit that change to my Git repositorylike I normally do in my regular workflow. And ask you to connectwill automatically pick up on that changeredeploy the content and make it available so thatI can make the change push it to GitHub and email thatdeveloper back and say it should be fixed now youshould now be able to receive Jason all right.So in conclusion, if you’reinterested in learning more about Plumber and the toolsthat we’ve talked about today. Here’s a whole listof resources you can go to visit the first list. The first resource listedhere is the Plumber website. And then a bunch ofdifferent Git repositories so the Plumber get a repository. If you run into like issuesor questions with the package you can go there to seeif that issue has already been submitted. Or you can submityour own issue. The second repositorylisted here is a collection of justexample APIs to build from.If you are lookingfor some examples to get started with a Plumberdeploy I briefly mentioned currently has theDigitalOcean functions for deploying DigitalOcean. This rapid R repositoryis something we use today. So that we could changethe user interface that we saw with Plumber. The material thatwe looked at today. So the example files andeverything like that. The slides willbe made available at the webinars GitHubrepository that’s listed here. So look for that to bepublished there shortly. And then the community. There are a community site hasa Plumber group or a Plumber tag that you can follow if you’reinterested in questions being asked about Plumber.If you havequestions on your own you’d like to askto the community. You can do. So there. And then finally,our state RStudio has our global conferencecoming up in January. Barrett, who is,again joined us today is going to be presenting someadditional new functionality with Plumber and asynchronousexecution at that. And that’s an entirely virtualconference that you can attend and I would encourageyou to attend as well, especially if you’re interestedin asynchronous Plumber execution as thatis what’s buried is going to bespeaking about there. And then finally, just thankyou again for your time, for joining us today. I appreciate thisopportunity to present on Plumber and allthe new features that we’ve added to the package. We’re really excited aboutthe work that’s being done. If you have questions, if youhave things you want to discuss feel free to reach out. Reach out on our your community. I monitor that sitequite frequently. And respond to questionsor comments at posts there. And again, just from allof us here in RStudio. Thank you very muchfor joining us today.I think we’ve got a couple ofminutes here for questions. And so I’m going to pause andpull open the questions here to see if there’s anythingthat remains to be answered. There’s a questionthat came in that says, are there anysuggestions suggested tools or packages tointegrate security to secure access to API. Really good question. There is not anything unlikethe R package standpoint that I’ve really tested. I know that a while ago. And I’m trying to remember thename I came across a group that was building an Rpackage that the goal was to provide securitysome security functionality for Plumber API. I haven’t revisitedthat in quite a while. And in fact like Isaid, I don’t quite remember what the name ofthat repository or package was going to be.So I don’t know wherethings ended up there. I don’t know howfar that made it. Because of Plumber’sflexibility. You can handle like securityand authentication and things like that yourself. It becomes I don’t wantto say tricky, but it just you have to thinkcarefully about it to make sure that youdon’t leave yourself open to like securityvulnerabilities but you could certainly forexample, have a filter in place that says, look, if auser if a request is made without a specific headeror without a specific token then that doesn’t getpassed through that request gets rejected and for isout or something like that. You can certainly do thaton your own within Plumber but there’s no like Isaid, to my knowledge, is there’s no kind of outof the box package or tool that does that for you.That being said,our RStudio Connect does have an API keybased authentication. So if I, for example, take my. Let me just come backhere for a minute. If I take my littleAPI here and set it. And instead said, look, youhave to log in to view this, then that would require usersto submit a request with an API key as one of their headersand ask you don’t connect will verify that key andverify that the key belongs to a user that is allowedpermission to this API. And then pass that through. But again, that’ssomething that’s enforced on theNarciso connect level and not being enforced at theplumbing level Plumber is still just responding asusual ask you to connect is acting as thegatekeeper in front of it to filter out requeststhat are invalid.Great question. Can you completely customizethe look of the API that makes it lookprettier for users or are you stuck withthe template you showed. So those template. So we looked at swaggerand we looked at rapid arc. Those are based off of theopen API specification. Basically what those do. Those are frameworksthat take that open API specification, which isbasically that big yellow file, we looked at. And then generate aUI layer on top of it. And then the nicething about Plumber is it connects the UI layerto the underlying our process that’s running the API. Right So I’ve got this nicedirect link between OK, I’ve got this API running. And now I can interact withit like we see here right.I’ve got this nicesort of UI layer. I’m like, how do I havea lot of experience with each of thoseindividual pieces. So for example, we’reusing rapid eye care right. I don’t know how customizable. The look and feel of thisparticular framework is there. I do know that there area handful of frameworks that exist today. And there’s kind of morebeing created almost to feel like all the time. And so it’s possiblethat some may provide additional flexibility. I don’t I don’tentirely know the answer to that particular question. Can Plumber dictate rate limits. Another excellent question. Not out of the box. There’s nothingspecific in Plumber where you can specify like anat a rate limit tag or setting or anything like that.You could build inrate limiting yourself. It would require someadditional legwork because you’d have to havesome way of identifying know how are you ratelimiting or you rate limiting to a specific user. And then you’d havesome sort of cash rate. Keep track of whattheir rate limited or what their currentconsumption is. And then check thatconsumption against the limit that they’ve beengiven and provide messaging around that limit. You can build allthat and apply it. I’m not saying thatit’s not possible. I’m just saying there’snot anything built in that allowsyou to rate limit, but you could certainlybuild that functionality into your eyes by usingtools, the Plumber provides. OK Yes OK. So once deployed how doyou submit the new data to the model API. It seems unwieldyto just like passing a giant browser you are. Totally correct. Right in most cases,these APIs I’m going to be interactedwith the web browser. But rather somebody else.So you maybe you’ve got afront end developer that’s writing JavaScript oryou’ve got a mobile app developer developing and sweptor whatever the case is all of these frameworks have theirown libraries packages tools for building andsending API requests and those developerswould often do that. So for example, as an ruser the httr our package is what you would useto build a request. So if you were an R usersubmitting a request to you. I could submit arequest to this API. We just built right. I could use the httr package tobuild that request, including the new data that’s going to bepassed to the predict end point and send that request over.And then the responsewill come back and I could workwith it from there. So in most cases,the nice thing is when you start talking aboutAPI as most developers will understand how towork with that. And they’ll havetheir own framework that they use forsending requests. And the nice thingis those requests will all be formattedthe same way because http is a standard format. So your API. It doesn’t matterif the request is coming from in ourscript using httr if it’s coming from Pythonusing the request module.It doesn’t matter if it’s comingfrom going it doesn’t matter. It doesn’t matter. The request comes in theAPI executes and returns the response. And in lots of frameworkshave their own way of making those requests OK. Last question. I’ll answer here in thelast minute that we’ve got. What is the advantage of usingan API versus shiny apps. Really good question. Shiny applicationsare really best suited for like human interaction. Right So if I’ve gota shiny application. I’m going to come in here. And I’m going to interact withit as an individual right. As API. It’s not as if theyexpected that users are going to come in hereand interact with the API. This way they might do ita little bit right just to check some things,but in most cases, an APIs going to be interactedwith programmatically.So I might build a Plumber API. So that another tool orthat my company developed can use my model or can reusethe business logic that I’ve implemented in my API. I might use a shiny app. So that another usercan view my model. If that makes sense, right. So that the distinctionhere is who’s the end user. In one case for shinyit’s another human that’s interacting withmy application for API. It’s most likeanother tool that’s interacting with myapplication or with my API..
James Blaire & Barret Schloerke | Integrating R With Plumber APIs | RStudio (2020)
James Blaire & Barret Schloerke | Integrating R With Plumber APIs | RStudio (2020)
James Blaire & Barret Schloerke | Integrating R With Plumber APIs | RStudio (2020)
James Blaire & Barret Schloerke | Integrating R With Plumber APIs | RStudio (2020)
James Blaire & Barret Schloerke | Integrating R With Plumber APIs | RStudio (2020)