Google Cloud Platform (GCP) plugin

Monitor your GCP environment, including GKE, Hosts and more.

How to add the Google Cloud Platform plugin

Prerequisites

In the GCP console select your project and check that the following are enabled:

VMware Engine API, see GCP - Enabling the VMware Engine
Google Kubernetes Engine (GKE), see GCP - Enabling the Artifact Registry and Google Kubernetes Engine APIs
Cloud Monitoring API, see GCP - Enabling the Monitoring API

To view cost information in SquaredUp using the Cost data stream check the following:

Export your billing data to Google BigQuery: GCP - Export Cloud Billing data to BigQuery
Enable BigQuery API, see GCP - Enable the BigQuery API

GCP Service Account Configuration

Create a new Service Account, or edit an existing account.
See GCP - Creating and managing service accounts
Ensure the account has the role Viewer by adding the role Basic > Viewer.
To view cost information, the account also needs to have BigQuery Data Viewer role.
See GCP - Grant a single role
Create a new key for the Service Account using the key type JSON. Download the JSON file as you will need to copy information from this JSON file when adding the plugin next.
See GCP - Creating service account keys
Make sure to store the key file securely, because it can be used to authenticate as your service account.

Add the Google Cloud Platform (GCP) plugin to SquaredUp

In SquaredUp browse to Settings > Plugins > Add plugin and search for the plugin.
Open the JSON file that you downloaded when creating the key.
Copy and paste the clientEmailfrom the JSON file into the plugin form.
Copy and paste the private_key from the JSON file into the plugin form (everything between the quotes).
Copy and paste the project_ids from the JSON file into the plugin form.
Optionally, to use the Cost data stream, copy and paste in the billingProjectId, billingDataSetName , and billingTableName.
Leave the checkbox for Automatically create a workspace ticked, this will create a new workspace for this plugin containing out of the box dashboards and scopes.

Optionally, select whether you would like to restrict access to this plugin instance. By default, restricted access is set to off.

Click Test and save to validate the plugin configuration.
Testing outcomes
- Testing passed – a success message will be displayed and then the configuration will be saved.
- Testing passed with warnings – warnings will be listed and potential fixes suggested. You can still use the plugin with warnings. Select Save with warnings if you believe that you can still use the plugin as required with the warnings listed. Alternatively, address the issues listed and then select Rerun tests to validate the plugin configuration again. If the validation now passes, click Save.
- Testing Failed – errors will be listed and potential fixes suggested. You cannot use the plugin with errors. You are able to select Save with errors if you believe that a system outside of SquaredUp Cloud is causing the error that you need to fix. Alternatively, address the issues listed and then select Rerun tests to validate the plugin configuration again. If the validation now passes, click Save.
You can edit any plugin configurations at any time from Settings > Plugins.

Using the Google Cloud Platform (GCP) plugin

data streams

There are three types of data streams available for this plugin:

Data streams installed with the plugin, see Tips for using the GCP Cost data stream
Configurable data streams, see Using Configurable data streams with the GCP plugin
Custom data streams, see Using Custom data streams with the GCP plugin

Using Configurable data streams with the GCP plugin

Google Cloud Platform - BigQuery

The GCP plugin allows you to set up configurable data streams for any kind of query for Google BigQuery.

Scope to an object that supports Google BigQuery.
Select + Google Cloud Platform - BigQuery from the data stream list.
Enter a Display Name for the new data stream.
Query:
Enter a BigQuery query.
Project ID:
Specify the project that you would like to run the BigQuery against.
Click Save.

Google Cloud Platform - MQL Query

The GCP plugin allows you to set up configurable data streams for any kind of query for Monitoring Query Language (MQL). Use the + MQL Query data stream to enter your own MQL queries into SquaredUp Cloud. You are able to query any object from your Google Cloud, even if they are not imported in the SquaredUp Cloud graph. See Google Cloud: Using Monitoring Query Language for more information about MQL.

Either scope to an object that supports MQL query, scope to a project, or scope to the GCP plugin instance itself.
Select + MQL Query from the data stream list.
Enter a Display Name for the new data stream.
Project:
Select the project that you would like to run the MQL against from the dropdown.
You will not need to select a project if you have already selected one in the scope.
MQL Query:
Enter a MQL query.
Mustache parameters are only supported if you have scoped to a GCP object and selected the + MQL Query. Mustache parameters are not supported if you have scoped to a project or the plugin instance itself.
See MQL query examples: below for more information about queries.
Supports mustache parameters supplied in an array
A mustache parameter is a dynamic value, the actual value will be inserted from the properties of the object(s) in scope.
This data stream supplies scoped objects in an array for mustache parameters. When there are multiple objects in scope this plugin will send the query once with all the objects in an array.
When the scoped objects are supplied in an array the normal mustache syntax, for example {{name}}, must be contained between {{#.}} and {{/.}} (the full-stop indicates that the whole object should be used, in this case the array of objects in scope).
For example, a query where clause might look like:
Copy
```
| where ComputerName in (  {{#.}}    '{{name}}',    {{/.}} '' )
```
- The {{#.}} and {{/.}} indicate that what is contained within is expanded for each element in the array of objects.
- You can use properties of objects and write them in between curly braces e.g {{name}} to use them as mustache parameters. For example, if objects of type "host" have a property called name, you can use {{name}}. This will resolve {{name}} to the value of the name property of the different "host" objects used in the scope.
- '{{name}}', means that the name property is expanded inside single-quotes with a trailing comma.
- The trailing single quotes '' are necessary to stop the query being rejected because a trailing comma is disallowed.
- Whenever you use mustache parameters, you need to use a scope of objects that contain the property you're referencing.
Automatically apply dashboard timeframe:
By default, Automatically apply dashboard timeframe will be selected.
The dashboard timeframe is the current timeframe setting for a dashboard. Users can change the dashboard timeframe to see data for a different time span, for example, instead of showing data from "the last 12 hours" it can be changed to show data from "the last 7 days".
Tiles can be configured to:
- Use dashboard timeframe (default). For these tiles the data shown will change when the user changes the dashboard timeframe.
- Use a fixed timeframe from the options available. These tiles show a clock icon and hovering shows the fixed timeframe configured. The data will not change when the dashboard timeframe is changed.
Tip: Indicate with the name of a tile if the tile's timeframe can be changed. For example, naming a tile "Performance during the last week" tells users that this tile always shows data for the last week. Naming a tile just "Performance" indicates to users that changing the dashboard timeframe will change the data.
If you do not wish to use a dashboard timeframe, you can set the timeframe in the MQL query by adding within to the query. If you deselect Automatically apply dashboard timeframe and do not specify a timeframe in the MQL query, the query will fail. To add a timeframe to a query, simply type | within (timeframe) at the end of your query. For example, to add a timeframe of 24 hours you enter: | within (24h)
Click Save.

MQL query examples:

MQL query not using dashboard timeframe - CPU usage time in the last 20 minutes in the selected project:

Copy

fetch gce_instance | metric 'compute.googleapis.com/instance/cpu/usage_time' | within (20m)

MQL query not using dashboard timeframe - authentication events count in the last 24 hours:

Copy

fetch iam_service_account
| metric 'iam.googleapis.com/service_account/authn_events_count'
| align rate(1m)
| every 1m | within (24h)

MQL query using dashboard timeframe - uploaded bytes in BigQuery data set:

Copy

fetch bigquery_dataset
| metric 'bigquery.googleapis.com/storage/uploaded_bytes'
| align rate(1m)
| every 1m

MQL query scoped to Hosts and using mustache syntax - CPU usage time using dashboard timeframe:

Copy

fetch gce_instance
            | metric 'compute.googleapis.com/instance/cpu/usage_time'
            | filter (metadata.system_labels.name == '{{#.}}{{name}}{{/.}}')
            | align rate(1m)
            | every 1m

Monitor Metric

Scope to an object that supports Monitor Metric.
If you are scoping to multiple objects, we suggest you only scope to objects of the same type. For example, the data will probably not be displayed in a helpful way if you were to select both Host objects and SQL objects in the same scope.
Select + Monitor Metric from the data stream list.
Enter a Display Name for the new data stream.
Metric Name:
Select the desired metric from the dropdown. The dropdown options will automatically update depending on the object that you have scoped to. For example, scope to a Host and select CPU usage from the Metric Name dropdown.
Click Save.

Using Custom data streams with the GCP plugin

Add one or more custom data streams in SquaredUp.

Go to Settings > Data Streams.
Click Add new Data Stream.
Enter a display name for your Data Stream.
Note: The display name is the name that you use to identify your Data Stream in SquaredUp. It has no technical impact and doesn't need to be referenced in the Data Stream's code.
Choose the plugin this Data Stream is for.
After you've chosen the plugin a new field Data Source appears.
Data source and code:
To find out which data source to select and get code examples for the Code field, see the help below.
Click Save to save your Data Stream.

Create generic scoped custom data stream for BigQuery

Which data source do I have to select from the dropdown?

BigQuery (Scoped)

Code example:

Copy

{
  "name": "exampleScopedBigQuery",
  "dataSourceConfig": {
    "query": "SELECT project.name, service.description FROM `<project_id>.<data_set_name>.<table_name>` WHERE project.id in ('{{sourceId}}') GROUP BY 1,2 ORDER BY 1,2;"
  },
  "rowPath": [],
  "matches": {
    "sourceType.0": {
      "type": "oneOf",
      "values": [
        "GCP Project"
      ]
    }
  },
      "metadata": [
        { "name": "name", "displayName": "Project","shape": "string", "role": "label" },
        { "name": "description", "displayName": "Service", "shape": "string" },
      ]
}

Parameters

name

Mandatory

The internal name of the data stream. Can be used the refer to this data stream in a tile's JSON instead of using the data stream's internal ID.

useTimeFrame

Optional

You can add "useTimeFrame": true to the dataSourceConfig to use the dashboard timeframe, and inject @startDate and/or @endDate parameters into your query and these will be passed to the dashboard timeframe.

For example:

Copy

{
    "name": "exampleBigQuery",
    "dataSourceConfig": {
        "useTimeFrame": true,
        "query": "SELECT <column_1> FROM `<project_id>.<data_set_name>.<table_name>` WHERE <dateColumn> BETWEEN @startDate AND @endDate"
    },
    "rowPath": [],
    "matches": "none",
    "metadata": []
}

query

Mandatory

Insert your query here. The query must be written all in one line. The query supports mustache parameters (in between curly braces e.g {{name}}). Mustache must refer to the property names of a GCP object that is the scope of the query. In the example we scope a query by GCP project. The query will execute for each object in the scope and return all results.

rowPath

Optional

Depends on the data returned from your query. Enter the path to the row you want to use.

matches

Matches should be populated appropriately to the type of objects in the scope, that you want this data stream to work for.

Note: Defining the matches parameter is mandatory.

With the matches parameter you define for which objects the data stream will be shown in the UI. It works like this:
When you configure a tile, you have to choose its scope. If this scope contains objects you specified here in the matches parameter, the data stream will be shown in the UI under Data Streams. If the scope doesn't contain objects specified here, the data stream will be hidden.
This keeps the UI clean and simple since you'll only see the data stream when it's relevant for the scope you chose. As a best practice you should limit the data stream to objects that make sense for the specific use case of this data stream.

Format for matches:

Copy

//If you want to specifiy only one value of an object property//
"matches": {
    "ObjectProperty": {
        "type": "equals",
        "value": "ValueOfTheObjectProperty"
    }
},
 
//If you want to specifiy multiple values for an object property//
"matches": {
    "ObjectProperty": {
         "type": "oneOf",
        "values": ["ValueOfTheObjectProperty1", "ValueOfTheObjectProperty2", "ValueOfTheObjectProperty3"]
    }
},

Example for limiting a data stream to objects:

If you are using multiple values for the object property, you can decide if you want the data stream to be visible for objects that match all of the criteria or at least one of the criteria.

Lets say you have two values you want objects to have in order for the data stream to be visible for them:

a SourceName property with the value AppDynamics (meaning objects that come from the AppDynamics plugin)
a type property with the value app (meaning application objects)

If you want the data stream to be visible only for objects that match both of the criteria, your code would look like this:

Copy

  "matches": {
    "sourceName": {
      "type": "equals",
      "value": "AppDynamics"
    },
    "type": {
      "type": "equals",
      "value": "app"
    }
  },

If you want the data stream to be visible for objects that match at least one of the criteria, your code would look like this:

Copy

"matches": [
    {
        "sourceName": {
            "type": "equals",
            "value": "AppDynamics"
        }
    },
    {
        "type": {
            "type": "equals",
            "value": "app"
        }
    }
]

Note: If you run into errors when configuring the matches parameter, check if you're dealing with a global data source.

metadata

Optional, but recommended

The metadata parameters are used to describe columns in order to tell SquaredUp what to do with them. You can do multiple things with the metadata parameters:

Specify how SquaredUp should interpret the columns you return and - to an extent - how their content displayed. You do this by giving each column a shape.
The shape you assign to a column tells SquaredUp what the column contains (for example, a number, a date, a currency, a URL, etc.). Based on the shape SquaredUp decides how to display this column, for example to display a URL as a clickable link.
Filter out or just hide columns.
Only the columns you define in metadata will be returned in the results. This helps you to filter out columns you don't need. If you need the content of a column but don't want to display it, you can use the visible parameter.
Give columns a nicely readable display name.
Assign a specific role to columns
The role you assign to a column tells SquaredUp the purpose of the column. For example, if you have two different columns that contain numbers, you need to assign the role value to the column that contains the actual value you want to use in your visualization.

Note: If you don't specify any metadata, all columns will be returned and SquaredUp will do its best to determine which columns should be used for which purpose. If you're returning pretty simple data, for example just a string and a number, this can work fine. But if you're returning two columns with numbers it gets trickier for SquaredUp to figure out which one is the value and which one is just an ID or some other number.

Parameters:

Tip: Before you start specifying metadata, leave them empty at first and get all the raw data with your new data stream once. In order to do this, finish creating your custom data stream without metadata and create a tile with this data stream. The Table visualization will show you all raw data.
This will give you an overview about all columns and their content and help you decide which columns you need and what their shapes and roles should be. It's also essential for getting the correct column name to reference in the name parameter.
Use this information to go back to the data stream configuration and specifying the metadata.

`name`	Mandatory	Enter the name of the column you are referencing here. To find the name of a column, get the data from this data stream once without any metadata. See the tip above for how to do that. You'll see the column name when you hover over the column in the Table.
`displayName`	Optional	Here you can give the column a user-friendly name for the UI
`shape`	Recommended	The shape you assign to a column tells SquaredUp what the column contains (for example, a number, a date, a currency, a URL, etc.). Based on the shape SquaredUp decides how to display this column, for example to display a URL as a clickable link. Note: Please refer to the list of shapes below this table to see available shapes.
`role`	Recommended	The role you assign to a column tells SquaredUp the purpose of the column. For example, if you have two different columns that contain numbers, you need to assign the role `value` to the column that contains the actual value you want to use in your visualization. Note: Please refer to the list of roles below this table to see available roles.
`visible`	Optional	`true` or `false` Use this if you need a columns content but don't need to display the column itself. Example: Column A contains the full link to a ticket in your ticket system. Column B contains the ticket ID. You want to use the ticket ID as a label for the link, turning the long URL into a much nicer to read "Ticket 123". This is why you need the content of column B, to assign it as a label for column A. But since the URL is now displayed as the ticket ID, it would be redundant to still display column B. This is why you hide column B with `false`.

Tips for using the GCP Cost data stream

The Cost data stream is part of the GCP plugin and uses the GCP BigQuery API. The GCP plugin needs some additional configuration to enable the Cost data stream see How to add the Google Cloud Platform plugin

The GCP Cost data stream shows daily data.
By default the Cost data stream splits costs by service.
We recommend scopes used with the GCP Cost data stream contain only one type of object, for example, only accounts, projects or hosts. Mixed scopes may include duplicated data in the results, because a query is sent for each type of object in the scope, and the results are combined. For example, resulting data would show the cost of a host, as well as the cost of a project, even if the host is part of that project. It is useful to create a scope for your account(s), a scope of project(s) and a scope for hosts.

Use Cases for the GCP Cost data stream

Cloud KB LIVE.10.log-183548538.zip
4 KB Download
Cloud KB LIVE.10.log-183548538.zip
4 KB Download
Cloud KB LIVE.10.log-183548538.zip
4 KB Download
Cloud KB LIVE.10.log-183548538.zip
4 KB Download
Cloud KB LIVE.10.log-183548538.zip
4 KB Download
Cloud KB LIVE.10.log-183548538.zip
4 KB Download
Cloud KB LIVE.10.log-183548538.zip
4 KB Download
Cloud KB LIVE.10.log-183548538.zip
4 KB Download
Cloud KB LIVE.10.log-183548538.zip
4 KB Download
Cloud KB LIVE.10.log-183548538.zip
4 KB Download
Cloud KB LIVE.10.log-183548538.zip
4 KB Download
Cloud KB LIVE.10.log-183548538.zip
4 KB Download
Cloud KB LIVE.10.log-183548538.zip
4 KB Download
Cloud KB LIVE.10.log-183548538.zip
4 KB Download
Cloud KB LIVE.10.log-183548538.zip
4 KB Download
Cloud KB LIVE.10.log-183548538.zip
4 KB Download
Cloud KB LIVE.10.log-183548538.zip
4 KB Download
Cloud KB LIVE.10.log-183548538.zip
4 KB Download
Cloud KB LIVE.10.log-183548538.zip
4 KB Download
Cloud KB LIVE.10.log-183548538.zip
4 KB Download
Cloud KB LIVE.10.log-183548538.zip
4 KB Download
Cloud KB LIVE.10.log-183548538.zip
4 KB Download
Cloud KB LIVE.10.log-183548538.zip
4 KB Download

Link to workspace	User can link the plugin to any workspace they have at least Editor permissions for. Data from the plugin can then be viewed by anyone with any access to the workspace. User can share the plugin data with anyone they want. User cannot configure the plugin in any way, or delete it.
Full Control	User can change the plugin configuration, ACL, and delete the plugin.

Group by	Service (For each service...)
Aggregate column	Amount (...show me the combined amount...)
Aggregation type	Total (... in total.)

Group by	Object (For each host...)
Aggregate column	Amount (...show me the combined amount...)
Aggregation type	Total (... in total.)

Group by	Object (For the account(s) in scope...)
Aggregate column	Amount (...show me the combined amount...)
Aggregation type	Total (... in total.)

Google Cloud Platform (GCP) plugin

How to add the Google Cloud Platform plugin

Using the Google Cloud Platform (GCP) plugin

data streams

Using Configurable data streams with the GCP plugin

Google Cloud Platform - BigQuery

Google Cloud Platform - MQL Query

MQL query examples:

Monitor Metric

Using Custom data streams with the GCP plugin

Tips for using the GCP Cost data stream

Use Cases for the GCP Cost data stream

Related articles