justinking

Google Drive Integration? No problem.

Blog Post created by justinking Employee on Feb 27, 2017

Updated 14 Dec 2017 to modify instructions for Google Drive setup including creating a Project.

 

In our implementation projects, customers ask us to connect to all sorts of things. Of course many of those are cloud-based but increasingly we see use cases involving non-business application endpoints such as document storage. One of my recent projects integrated with Google Drive so I wanted to share some tips for connecting with the Drive API.

 

This post outlines the steps necessary to implement some key capabilities when configuring AtomSphere to interface to the Google Drive REST API. It covers the OAuth 2.0 setup and several use cases to read and write folders and files within Google Drive.

 

 Get the example process used below from the Process Library here.

 

 

Configuring the OAuth 2.0 Connection

OAuth 2.0 needs to be configured to be able to access user defined content from Google Drive. OAuth 2.0 is setup using the Google’s documentation which isn’t difficult but setting up the OAuth Access Token can be tricky. If you follow the directions provided by Google, you’ll generate a token that times out every hour. This obviously doesn't work nicely when you have a scheduled, unattended integration in AtomSphere so the solution is to use a Refresh Token that doesn’t time out. This step is easy to miss.

 

Configure Google Drive

First create a Google OAuth 2.0 account by going to the Google API Console. On the far left of the screen, you’ll see API and three icons. The second is for the Library and the last is for creating Credentials. Select the Library first, look for Drive API and select the Google Drive API button under the G Suite section.

 

 

At the top of the page you’ll see Google Drive API...click Enable.

 

 

In order to access your google drive folders/files, a project must be created. The next screen will ask you to create a project.

 

 

 

Click Create button and enter a name. Any name will do.

 

 

 

The Google Drive API screen will re-appear. Click Enable again and ensure the project that was created has been selected.

 

 

You should see an Enabling API spinning circle.

 

 

Once complete, you should see the following screen.

 

 

Next, configure the Credentials...

 

 

Once that is finished, go to the Credentials screen. At the top select the tab for “OAuth consent screen”. This is where you will select the Google account that will be using the AtomSphere connection. Add the google email address you will be using and a “Product name shown to users” then select “Save”. You can leave the remaining fields blank.

 

 

Go back on the Credentials screen and select the blue “Create credentials” button then select OAuth client ID.

 

 

On the Create client ID screen configure the following:

 

 

For Authorized redirect URIs, use the following value, replacing <your_account_id> with your AtomSphere Account ID:

https://platform.boomi.com/account/<your_account_id>/oauth2/callback

 

Select Create at the bottom of the page and the Client ID and Client Secret will appear on the screen. Save these as they will be used in the next section to configuration the HTTP Client connection. The Client ID will come appended with ".apps.googleusercontent.com".  Remove this string from the Client ID. 

 

Configure the HTTP Client Connection

Once Google is setup to access Google Drives to use OAuth 2.0 for making requests, we can configure the HTTP Client connection back in AtomSphere with the following values:

 

Your Connection should look like this:

 

 

Once your connection is configured, select the “Generate…” box next to Access Token. This will open a new window asking you to log into the Google account to authorize token generation. Select “Allow” when asked to allow the connector to “Have offline access”. A message will appear saying “Authorization Code received”. Go back to the AtomSphere tab and you will see a popup saying “Access Token generation successful”. Save the connection and you will be ready to access Google Drives.

 

Integrating with Google Drive

Now that we have the connection configured, let's use it in a simple example that demonstrates some of the common API calls to list, get, and create files. In this example, we're going to retrieve all the CSV files from a "monthly" folder (named like "2017-02" for example), produce a simple summary XML report based on those files, and then upload that report file to the same monthly folder in Drive.

 

The high level approach is:

  1. List the single folder for the current month to get its details, IDs, etc.
  2. List all the files in that folder.
  3. For each CSV file, retrieve the actual file contents.
  4. Summarize and combine the files.
  5. Create a new file in Drive.

 

The overall integration process looks like this:

 

 

 Get the example process from the Process Library here.

 

Let's take a closer look at each of those calls.

 

List Files/Folders

To access individual files or directories from Google Drive you need their respective IDs. We can use the List API to retrieve the list of files and folders along with their metadata.

 

The first List call specifies the specific "monthly" folder name and mime type to retrieve a folder on Google Drive. The URL looks like:

https://www.googleapis.com/drive/v3/files?corpus=user&q=name=%27<folder_name>%27%20and%20mimeType%20%3D%20%27application%2Fvnd.google-apps.folder%27

 

 

Breaking out the URI parameters:

  • corpus - Defines the source of the files to list. This can be either “user” or “domain”. In our case it’s user so we can access files owned by or shared to the user.
  • q - This is used to query for a specific file or folder. In the above example we are using name and mimeType to find a specific folder. There are other parameters that can be used to perform searches. As you can see in the example the query is URL encoded. Also, the name parameter is join to the mimetype with an “and” to continue the query.

 

The call above could also be used to list a specific file by passing the file name and MIME type for the file you want.

 

The second List function uses the folder ID retrieved in the first List and returns all the files and folders found in the folder specified.  That URL looks like:

https://www.googleapis.com/drive/v3/files?corpus=user&q=%20<folder_ID>%27%20in%20parents

 

 

The query parameter here is “in parents” which will return all files and folders in a parent directory. From here we can identify and get the files that we need from the directory.

 

Get Files

The IDs from the List can now be used to get the actual file content using the Get API. The URL looks like:

https://www.googleapis.com/drive/v3/files/<Google_file_ID>?alt=media

 

The ?alt=media parameter tells us to not send metadata but to send the files. As seen below, file ID is being passed in as a replacement variable in the Resource Path. This is set in the Parameters on the connector using the Profile Element of the JSON data returned from Google.

 

 

This will work for files using formats such as .csv, .doc, .xlxs, .ppt. etc. The Get API will not work for Google file types such as Google Docs, Google Spreadsheets, or any other document created by Google. To download those types of files, the Export API is used, which works similarly to the Get. The List function will also list the mimetype so a Decision shape can be used to potentially set up a Get and Export if both are being used.

 

Create Files with Metadata

The last piece of our example is to create the file on Google Drive. The first thing we are going to do is combine the CSV records into a single file, then use a Set Properties to create a name for the file. Both of those will be used in the Message shape where the payload for Google is pieced together before sending it to Google.  Below is the process flow used in the use-case to create a file on Google Drive:

 

 

There are three options for creating a file on Drive: Simple, Multipart, and Resumable uploads. The Simple upload will upload the file but without any metadata such as file type or even file name. The Resumable upload uses multiple sessions to upload the file so a file upload can be restarted if communications has been interrupted. In this example we'll use Multipart upload to create the file with metadata.

 

The URL looks like:

https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart

 

 

The uploadType parameter defines the type of upload to perform. Within the HTTP Client operation, add a Request Header “Content-Type” with of “multipart/related; boundary=BOOMI”. The boundary here is used to define the payload’s starting and ending position and also what separates the metadata from the file data within the payload.

 

The following used be used to create the payload being passed into the HTTP POST:

 

 

As you can see, the “boundary” set to “BOOMI” identifies the beginning and end of the file and splits the metadata from the file content. The metadata is defined by JSON where a name, mimeType and parent directory are being defined. The mimeType in this use case is going to be text/plain but can be changed to many other non-Google, .csv, .doc, .xlxs, .ppt. etc, file types. The name and parent folder/directory are being defined by Variables in this Message shape just before the HTTP Connector. The file name is a dynamic document property (or it can be statically) created in a Set Properties shape prior to the Message shape defined above. The parent folder ID is obtained from the dynamic process property captured at the beginning of the process. The JSON can be tailored as necessary based on need. Content Type is also defined in the file content section and changes can be made to reflect the file that is being passed in be the user. The XML report file content is passed in as the Current Data.

 

Note: It is important to remember when using a Message shape to pay close attention to double and single quotes. Because of the JSON data, the message must start with a single quote and also have single quotes around the Variables being added to the Message shape, as seen below. The following is the Message shape content for use in your integration.

 

'--BOOMI

Content-Type: application/json; charset=UTF-8

 

{

"name" : "'{1}'",

"mimeType": "text/plain",

"parents":["'{2}'"]

}

 

--BOOMI

Content-Type: text/plain

 

'{3}'

--BOOMI--

 

Justin King is an integration consultant for the Dell Boomi Professional Services team.

Outcomes