We mentioned the command-line program <sh:app>curl</sh:app> at the beginning of this chapter. <sh:app>curl</sh:app> is based on <libcurl></libcurl> so it provides ways to use many of <libcurl></libcurl>'s facilities via command-line flags and arguments. We can set options such as the User-Agent or the netrc.file and specify files and other inputs for Web requests on the command line. Since RCurl and <sh:app>curl</sh:app> share the same underlying software (<libcurl></libcurl>), they both can access many of the same features. There are many examples on the Web of using <sh:app>curl</sh:app> for making HTTP (HyperText Transfer Protocol) requests so sometimes it can be useful to understand how to translate those to R commands using RCurl functions. In this section, we will look at some examples and show the <sh:app>curl</sh:app> shell command and the equivalent R command.
Many of the <sh:app>curl</sh:app> command-line flags
have two forms <dash></dash> the longer, but more suggestive name
such as ----user-agent
and the simpler
to type ---A
.
The longer forms correspond reasonably closely to the names
of the curl options in RCurl.
When translating a shell command invoking curl,
it is good practice to consult the help page for curl
and map the short forms of the command line options to their longer counterpart,
at least mentally, and then map them to the corresponding RCurl option names.
<duncan>We seem to want to use PUT, but use POST. We don't use the up variable.</duncan> For example, we can upload the contents of a file to a <url></url> using <sh:app>curl</sh:app> with a command such as
curl -T filename -X PUT url
The ---X
identifies the type of HTTP (HyperText Transfer Protocol) request.
The ---T
flag specifies the file whose contents
we want to send.
We can do this in R, with
up = fileUpload(filename) httpPUT(url, upload = TRUE, readdata = CFILE(filename)@ref, infilesize = file.info(filename)[1, "size"])
Example 1. Comparing Command Line and RCurl Requests for GlobalGiving
As another example, we send a request to a <rest></rest> API provided by
GlobalGiving [???]
at www.globalgiving.org. It
explicitly sets two HTTP (HyperText Transfer Protocol) header fields <dash></dash>
'Accept'
and
'Content-Type'
. We will also
set the 'User-Agent'
field.
The command is
curl -H "Accept: application/xml" -H "Content-Type: application/xml" -A 'curl' -X GET url
where url
is
https://api.globalgiving.org/api/public/projectservice/all/projects/ids?api_key=YOUR_API_KEY
and you insert the appropriate key.
With RCurl, we can use getForm()
and
pass the <url></url> and parameter(s) as separate arguments rather than
having to paste our key to the <url></url>. We do this with uGG as
https://api.globalgiving.org/api/public/projectservice/all/projects/ids and
getForm(uGG, api_key = key, useragent = "RCurl", httpheader = c(Accept = "application/xml", "Content-Type" = "application/xml" ))
The ---A
in <sh:app>curl</sh:app> corresponds to
the useragent option in RCurl
and the two ---H
arguments
are combined into a vector in R and set via the httpheader option.
Example 2. Posting a Tweet with a <sh:app>curl</sh:app> Command vs. httpPOST()
We can post tweets programmatically via a POST
request.
Twitter [???]
provides a Web interface to show you what the request would be as a
<sh:app>curl</sh:app> command. (See https://dev.twitter.com/apps and the OAuth
tab for your
particular application.) For example, we post the tweet "Here
it is"
with
curl --request 'POST' 'https://api.twitter.com/1/statuses/update' --data 'Here_it_is=' --header 'Authorization: OAuth oauth_consumer_key=xx, oauth_nonce=45b55, oauth_signature_method=HMAC-SHA1, oauth_token=xxxx' --verbose
With RCurl, we can use
uT = 'https://api.twitter.com/1/statuses/update' httpPOST(uT, postfields = "Here it is", httpheader = c(Authorization = 'OAuth oauth_consumer_key=xx, oauth_nonce=45b55, oauth_signature_method=HMAC-SHA1, oauth_token=xxxx' ), verbose = TRUE )
Note that RCurl encodes the text ("Here it is"
) for us.
The ----request
flag is the same as the shorter ---X
flag for identifying the method of the request, e.g., POST
or PUT
.
One of the benefits of RCurl relative to <sh:app>curl</sh:app> is that we are working in R rather than the shell. This means we have a more flexible programming language and richer data structures. This makes it easier to do computations before and after the Web requests, that is we can can create the inputs from other computations, with data in R. We can also use the same curl handle across requests which makes it easier to manage state and content (e.g., cookies) across the requests.
Example 3.
curl -H "Content-Type: application/xml" -u admin:district -d @categoryCombos.xml "http://localhost:8081/master/api/metaData" -X POST -v
The ---H
corresponds to the httpheader option in a call to any of the
RCurl functions.
httpheader = c("Content-Type" = "application/xml")
The ---u
corresponds to userpwd, i.e.
userpwd = "admin:district"
The ---X
corresponds to either
customrequest = "POST"
or alternatively using the httpPOST() function directly.
The ---v
is for verbose.
The most complicated or least obvious flag in this this command line is ---d
.
This identifies the data to send as the body of the <http:op>POST</http:op> request.
The @ sign tells curl
that the argument is the name of a file
and to use the contents of that file to get the actual content of the request's body.
RCurl()
allows us to do deal with this in several ways.
We can read the contents of the file ourselves in R and pass that as the body of the
request using postfields, as in the twitter update example.
Alternatively, we can use readdata as in the example above.
So we can implement this with
httpPOST("http://localhost:8081/master/api/metaData", verbose = TRUE, httpheader = c("Content-Type" = "application/xml"), userpwd = "admin:district", postfields = paste(readLines("categoryCombos.xml"), collapse = "\n"))