Google Cloud Platform (GCP) plugin

Visit our website to see the data that you can access if you use this plugin to add the data source to SquaredUp:

Google Cloud Platform

Monitor your GCP environment, including GKE, Hosts and more.

How to add a Google Cloud Platform data source

Prerequisites

In the GCP console select your project and check that the following are enabled:

VMware Engine API, see GCP - Enabling the VMware Engine
Google Kubernetes Engine (GKE), see GCP - Enabling the Artifact Registry and Google Kubernetes Engine APIs
Cloud Monitoring API, see GCP - Enabling the Monitoring API

To view cost information in SquaredUp using the Cost data stream check the following:

Export your billing data to Google BigQuery: GCP - Export Cloud Billing data to BigQuery
Enable BigQuery API, see GCP - Enable the BigQuery API

GCP Service Account Configuration

Create a new Service Account, or edit an existing account.
See GCP - Creating and managing service accounts
Ensure the account has the role Viewer by adding the role Basic > Viewer.
To view cost information, the account also needs to have BigQuery Data Viewer role.
See GCP - Grant a single role
Create a new key for the Service Account using the key type JSON. Download the JSON file as you will need to copy information from this JSON file when adding the data source next.
See GCP - Creating service account keys
Make sure to store the key file securely, because it can be used to authenticate as your service account.

Add a Google Cloud Platform (GCP) data source to SquaredUp

To add a data source click on the + next to Data Sources on the left-hand menu in SquaredUp. Search for the data source and click on it to open the Configure data source page.
Open the JSON file that you downloaded when creating the key.
Copy and paste the clientEmailfrom the JSON file into the data source form.
Copy and paste the private_key from the JSON file into the data source form (everything between the quotes).
Copy and paste the project_ids from the JSON file into the data source form.
Optionally, to use the Cost data stream, copy and paste in the billingProjectId, billingDataSetName , and billingTableName.

Optionally, select whether you would like to restrict access to this data source instance. By default, restricted access is set to off.

Click Test and add to validate the data source configuration.
Testing outcomes
- Testing passed – a success message will be displayed and then the configuration will be saved.
- Testing passed with warnings – warnings will be listed and potential fixes suggested. You can still use the data source with warnings. Select Save with warnings if you believe that you can still use the data source as required with the warnings listed. Alternatively, address the issues listed and then select Rerun tests to validate the data source configuration again. If the validation now passes, click Save.
- Testing Failed – errors will be listed and potential fixes suggested. You cannot use the data source with errors. You are able to select Save with errors if you believe that a system outside of SquaredUp is causing the error that you need to fix. Alternatively, address the issues listed and then select Rerun tests to validate the data source configuration again. If the validation now passes, click Save.
You can edit any data source configurations at any time from Settings > Data Sources.
You can also add a data source from Settings > Data Sources > Add data source, but sample dashboards are not added when using this method.

Using the Google Cloud Platform (GCP) data streams

Data streams installed with the data source, see Tips for using the GCP Cost data stream

Write a custom data stream (advanced use) see Writing a custom data stream (advanced users)

Google Cloud Platform - BigQuery

The GCP data source allows you to set up configurable data streams for any kind of query for Google BigQuery.

In the tile editor, filter by the GCP data source, select BigQuery from the data stream list and then click Next.
Query:
Enter a BigQuery query.
Project ID:
Specify the project that you would like to run the BigQuery against.
Click Save.

Google Cloud Platform - MQL Query

The GCP data source allows you to set up configurable data streams for any kind of query for Monitoring Query Language (MQL). Use the + MQL Query data stream to enter your own MQL queries into SquaredUp. You are able to query any object from your Google Cloud, even if they are not indexed in the SquaredUp graph. See Google Cloud: Using Monitoring Query Language for more information about MQL.

In the tile editor, filter by the GCP data source, select MQL Query from the data stream list and then click Next.
You can either select the scoped MQL Query data stream or the global MQL Query data stream.
If you selected the scoped MQL Query data stream, select the objects you want to use and then click Next.
You do not need to do this if you have selected the global MQL Query data stream.
Project:
Select the project that you would like to run the MQL against from the dropdown.
You will not need to select a project if you have already selected one in the scope.
MQL Query:
Enter a MQL query.
Mustache parameters are only supported if you have scoped to a GCP object and selected the + MQL Query. Mustache parameters are not supported if you have scoped to a project or the data source instance itself.
See MQL query examples: below for more information about queries.
Supports mustache parameters supplied in an array
A mustache parameter is a dynamic value, the actual value will be inserted to replace the field in curly braces. For example, {{timeframe.start}} will insert the start time based on the timeframe configured within the tile, or {{name}} will insert the name of the object(s) in scope.
This data stream supplies scoped objects in an array for mustache parameters. When there are multiple objects in scope this data source will send the query once with all the objects in an array.
When the scoped objects are supplied in an array the normal mustache syntax, for example {{name}}, must be contained between {{#.}} and {{/.}} (the full-stop indicates that the whole object should be used, in this case the array of objects in scope).
For example, a query where clause might look like:
Copy
```
| where ComputerName in (  {{#.}}    '{{name}}',    {{/.}} '' )
```
- The {{#.}} and {{/.}} indicate that what is contained within is expanded for each element in the array of objects.
- You can use properties of objects and write them in between curly braces e.g {{name}} to use them as mustache parameters. For example, if objects of type "host" have a property called name, you can use {{name}}. This will resolve {{name}} to the value of the name property of the different "host" objects used in the scope.
- '{{name}}', means that the name property is expanded inside single-quotes with a trailing comma.
- The trailing single quotes '' are necessary to stop the query being rejected because a trailing comma is disallowed.
- Whenever you use mustache parameters, you need to use a scope of objects that contain the property you're referencing.
Automatically apply dashboard timeframe:
By default, Automatically apply dashboard timeframe will be selected.
The dashboard timeframe is the current timeframe setting for a dashboard. Users can change the dashboard timeframe to see data for a different time span, for example, instead of showing data from "the last 12 hours" it can be changed to show data from "the last 7 days".
Tiles can be configured to:
- Use dashboard timeframe (default). For these tiles the data shown will change when the user changes the dashboard timeframe.
- Use a fixed timeframe from the options available. These tiles show a clock icon and hovering shows the fixed timeframe configured. The data will not change when the dashboard timeframe is changed.
Tip: Indicate with the name of a tile if the tile's timeframe can be changed. For example, naming a tile "Performance during the last week" tells users that this tile always shows data for the last week. Naming a tile just "Performance" indicates to users that changing the dashboard timeframe will change the data.
If you do not wish to use a dashboard timeframe, you can set the timeframe in the MQL query by adding within to the query. If you deselect Automatically apply dashboard timeframe and do not specify a timeframe in the MQL query, the query will fail. To add a timeframe to a query, simply type | within (timeframe) at the end of your query. For example, to add a timeframe of 24 hours you enter: | within (24h)
Click Save.

MQL query examples:

MQL query not using dashboard timeframe - CPU usage time in the last 20 minutes in the selected project:

Copy

fetch gce_instance | metric 'compute.googleapis.com/instance/cpu/usage_time' | within (20m)

MQL query not using dashboard timeframe - authentication events count in the last 24 hours:

Copy

fetch iam_service_account
| metric 'iam.googleapis.com/service_account/authn_events_count'
| align rate(1m)
| every 1m | within (24h)

MQL query using dashboard timeframe - uploaded bytes in BigQuery data set:

Copy

fetch bigquery_dataset
| metric 'bigquery.googleapis.com/storage/uploaded_bytes'
| align rate(1m)
| every 1m

MQL query scoped to Hosts and using mustache syntax - CPU usage time using dashboard timeframe:

Copy

fetch gce_instance
            | metric 'compute.googleapis.com/instance/cpu/usage_time'
            | filter (metadata.system_labels.name == '{{#.}}{{name}}{{/.}}')
            | align rate(1m)
            | every 1m

Monitor Metric

In the tile editor, filter by the GCP data source, select Monitor Metric from the data stream list and then click Next.
Select the objects that you want to use and then click Next.
If you are selecting multiple objects, we suggest you only scope to objects of the same type. For example, the data will probably not be displayed in a helpful way if you were to select both Host objects and SQL objects in the same scope.
Metric Name:
Select the desired metric from the dropdown. The dropdown options will automatically update depending on the object that you have scoped to. For example, scope to a Host and select CPU usage from the Metric Name dropdown.
Click Save.

Workflow

Just like other services, Workflow is also supported by the GCP data source. You can monitor Workflow executions, execution times, and finished execution counts.

To set up a data stream for Workflow:

In the tile editor, filter by the GCP data source or select Workflow from the scope, then select your desired metrics (Workflow executions, Execution times, Finished execution count) from the list of data streams, and then click Next.
Select the Workflow you wish to scope to, then click Next.
Optionally, select a Timeframe or data shaping.
Click Save.

Build

The GCP data source also supports monitoring Builds from the Build Trigger service.

To set up a data stream for Build:

In the tile editor, filter by the GCP data source or select Build Trigger from the scope, then select Builds from the list of data streams, and then click Next.
Select the Build Trigger you wish to scope to, then click Next.
Optionally, select a Timeframe or data shaping.
Click Save.

PubSub

For PubSub service, you can monitor Publish requests, Publish message size, Oldest unacked message age, and Unacked messages.

To set up a data stream for PubSub:

In the tile editor, filter by the GCP data source or select Topic or Subscription from the scope (depending on what you want to monitor), then select your desired metrics from the list of data streams, and then click Next.
Select the Topic or Subscription you wish to scope to, then click Next.
Optionally, select a Timeframe or data shaping.
Click Save.

Firestore

The Firestore Database service is now supported by the GCP data source. You can set up configurable data streams for Structured query and Document.

To set up a data stream for Firestore:

In the tile editor, filter by the GCP data source or select Database from the scope, then select Structured query or Document from the list of data streams, and then click Next.
Select the Database you wish to scope to, then click Next.
Optionally, select a Timeframe or data shaping.
Click Save.

Writing a custom data stream (advanced users)

Add one or more custom data streams in SquaredUp.

Go to Settings > Advanced > Data Streams.
Click Add new Data Stream.
Enter a display name for your Data Stream.
Note: The display name is the name that you use to identify your Data Stream in SquaredUp. It has no technical impact and doesn't need to be referenced in the Data Stream's code.
Choose the Data Source this Data Stream is for.
After you've chosen the data source a new field Entry Point appears.
Entry point and code:
What is an entry point?
Each data stream uses an entry point, which can either be global (unscoped) or scoped, and this determines whether the data stream uses the tile scope.
Data streams can be either global or scoped:
- Global data streams are unscoped and return information of a general nature (e.g. "Get the current number of unused hosts").
- A scoped data stream gets information relevant to the specific set objects supplied in the tile scope (e.g. "Get the current session count for these hosts").
To find out which entry point to select and get code examples for the Code field, see the help below.
Click Save to save your Data Stream.

Create generic scoped custom data stream for BigQuery

Which entry point do I have to select from the dropdown?

BigQuery (Scoped)

Code example:

Copy

{
  "name": "exampleScopedBigQuery",
  "dataSourceConfig": {
    "query": "SELECT project.name, service.description FROM `<project_id>.<data_set_name>.<table_name>` WHERE project.id in ('{{sourceId}}') GROUP BY 1,2 ORDER BY 1,2;"
  },
  "rowPath": [],
  "matches": {
    "sourceType.0": {
      "type": "oneOf",
      "values": [
        "GCP Project"
      ]
    }
  },
      "metadata": [
        { "name": "name", "displayName": "Project","shape": "string", "role": "label" },
        { "name": "description", "displayName": "Service", "shape": "string" },
      ]
}

Parameters

name

Mandatory

The internal name of the data stream. Can be used the refer to this data stream in a tile's JSON instead of using the data stream's internal ID.

useTimeFrame

Optional

You can add "useTimeFrame": true to the dataSourceConfig to use the dashboard timeframe, and inject @startDate and/or @endDate parameters into your query and these will be passed to the dashboard timeframe.

For example:

Copy

{
    "name": "exampleBigQuery",
    "dataSourceConfig": {
        "useTimeFrame": true,
        "query": "SELECT <column_1> FROM `<project_id>.<data_set_name>.<table_name>` WHERE <dateColumn> BETWEEN @startDate AND @endDate"
    },
    "rowPath": [],
    "matches": "none",
    "metadata": []
}

query

Mandatory

Insert your query here. The query must be written all in one line. The query supports mustache parameters (in between curly braces e.g {{name}}). Mustache must refer to the property names of a GCP object that is the scope of the query. In the example we scope a query by GCP project. The query will execute for each object in the scope and return all results.

rowPath

Optional

Depends on the data returned from your query. Enter the path to the row you want to use.

matches

Matches should be populated appropriately to the type of objects in the scope, that you want this data stream to work for.

Note: Defining the matches parameter is mandatory.

With the matches parameter you define for which objects the data stream will be shown in SquaredUp. It works like this:
When you configure a tile, you have to choose its scope. If this scope contains objects you specified here in the matches parameter, the data stream will be shown in SquaredUp under Data Streams. If the scope doesn't contain objects specified here, the data stream will be hidden.
This keeps things clean and simple since you'll only see the data stream when it's relevant for the scope you chose. As a best practice you should limit the data stream to objects that make sense for the specific use case of this data stream.

Format for matches:

Copy

//If you want to specify only one value of an object property//
"matches": {
    "ObjectProperty": {
        "type": "equals",
        "value": "ValueOfTheObjectProperty"
    }
},
 
//If you want to specify multiple values for an object property//
"matches": {
    "ObjectProperty": {
         "type": "oneOf",
        "values": ["ValueOfTheObjectProperty1", "ValueOfTheObjectProperty2", "ValueOfTheObjectProperty3"]
    }
},

Example for limiting a data stream to objects:

If you are using multiple values for the object property, you can decide if you want the data stream to be visible for objects that match all of the criteria or at least one of the criteria.

Lets say you have two values you want objects to have in order for the data stream to be visible for them:

a SourceName property with the value AppDynamics (meaning objects that come from the AppDynamics data source)
a type property with the value app (meaning application objects)

If you want the data stream to be visible only for objects that match both of the criteria, your code would look like this:

Copy

  "matches": {
    "sourceName": {
      "type": "equals",
      "value": "AppDynamics"
    },
    "type": {
      "type": "equals",
      "value": "app"
    }
  },

If you want the data stream to be visible for objects that match at least one of the criteria, your code would look like this:

Copy

"matches": [
    {
        "sourceName": {
            "type": "equals",
            "value": "AppDynamics"
        }
    },
    {
        "type": {
            "type": "equals",
            "value": "app"
        }
    }
]

Note: If you run into errors when configuring the matches parameter, check if you're dealing with a global entry point.

Data streams can be either global or scoped:

Global data streams are unscoped and return information of a general nature (e.g. "Get the current number of unused hosts").
A scoped data stream gets information relevant to the specific set objects supplied in the tile scope (e.g. "Get the current session count for these hosts").

metadata

Optional, but recommended

The metadata parameters are used to describe columns in order to tell SquaredUp what to do with them. You can do multiple things with the metadata parameters:

Specify how SquaredUp should interpret the columns you return and - to an extent - how their content displayed. You do this by giving each column a shape.
The shape you assign to a column tells SquaredUp what the column contains (for example, a number, a date, a currency, a URL, etc.). Based on the shape SquaredUp decides how to display this column, for example to display a URL as a clickable link.
Filter out or just hide columns.
Only the columns you define in metadata will be returned in the results. This helps you to filter out columns you don't need. If you need the content of a column but don't want to display it, you can use the visible parameter.
Give columns a nicely readable display name.
Assign a specific role to columns
The role you assign to a column tells SquaredUp the purpose of the column. For example, if you have two different columns that contain numbers, you need to assign the role value to the column that contains the actual value you want to use in your visualization.

Note: If you don't specify any metadata, all columns will be returned and SquaredUp will do its best to determine which columns should be used for which purpose. If you're returning pretty simple data, for example just a string and a number, this can work fine. But if you're returning two columns with numbers it gets trickier for SquaredUp to figure out which one is the value and which one is just an ID or some other number.

Parameters:

Tip: Before you start specifying metadata, leave them empty at first and get all the raw data with your new data stream once. In order to do this, finish creating your custom data stream without metadata and create a tile with this data stream. The Table visualization will show you all raw data.
This will give you an overview about all columns and their content and help you decide which columns you need and what their shapes and roles should be. It's also essential for getting the correct column name to reference in the name parameter.
Use this information to go back to the data stream configuration and specifying the metadata.

`name`	Mandatory	Enter the name of the column you are referencing here. To find the name of a column, get the data from this data stream once without any metadata. See the tip above for how to do that. You'll see the column name when you hover over the column in the Table.
`displayName`	Optional	Here you can give the column a user-friendly name
`shape`	Recommended	The shape you assign to a column tells SquaredUp what the column contains (for example, a number, a date, a currency, a URL, etc.). Based on the shape SquaredUp decides how to display this column, for example to display a URL as a clickable link. Note: Please refer to the list of shapes below this table to see available shapes.
`role`	Recommended	The role you assign to a column tells SquaredUp the purpose of the column. For example, if you have two different columns that contain numbers, you need to assign the role `value` to the column that contains the actual value you want to use in your visualization. Note: Please refer to the list of roles below this table to see available roles.
`visible`	Optional	`true` or `false` Use this if you need a columns content but don't need to display the column itself. Example: Column A contains the full link to a ticket in your ticket system. Column B contains the ticket ID. You want to use the ticket ID as a label for the link, turning the long URL into a much nicer to read "Ticket 123". This is why you need the content of column B, to assign it as a label for column A. But since the URL is now displayed as the ticket ID, it would be redundant to still display column B. This is why you hide column B with `false`.

Tips for using the GCP Cost data stream

The Cost data stream is part of the GCP data source and uses the GCP BigQuery API. The GCP data source needs some additional configuration to enable the Cost data stream see How to add a Google Cloud Platform data source

The GCP Cost data stream shows daily data.
By default the Cost data stream splits costs by service.
We recommend scopes used with the GCP Cost data stream contain only one type of object, for example, only accounts, projects or hosts. Mixed scopes may include duplicated data in the results, because a query is sent for each type of object in the scope, and the results are combined. For example, resulting data would show the cost of a host, as well as the cost of a project, even if the host is part of that project. It is useful to create a scope for your account(s), a scope of project(s) and a scope for hosts.

Use Cases for the GCP Cost data stream

Cloud KB LIVE.10.log-183548538.zip
4 KB Download

Link to workspace	User can link the data source to any workspace they have at least Editor permissions for. Data from the data source can then be viewed by anyone with any access to the workspace. User can share the data source data with anyone they want. User cannot configure the data source in any way, or delete it.
Full Control	User can change the data source configuration, ACL, and delete the data source.

Group by	Service (For each service...)
Aggregate column	Amount (...show me the combined amount...)
Aggregation type	Total (... in total.)

Group by	Object (For each host...)
Aggregate column	Amount (...show me the combined amount...)
Aggregation type	Total (... in total.)

Group by	Object (For the account(s) in scope...)
Aggregate column	Amount (...show me the combined amount...)
Aggregation type	Total (... in total.)