Cloudlist Providers - ProjectDiscovery Documentation

This guide offers insights into each supported provider, enabling you to leverage Cloudlist’s capabilities to their fullest extent for comprehensive asset visibility and control.

Major Cloud Providers

AWS (Amazon Web Services)

Supported AWS Services:

Example Config: Amazon Web Services can be integrated by using the following configuration block.

- provider: aws # provider is the name of the provider
  # id is the name defined by user for filtering (optional)
  id: staging
  # aws_access_key is the access key for AWS account
  aws_access_key: $AWS_ACCESS_KEY
  # aws_secret_key is the secret key for AWS account
  aws_secret_key: $AWS_SECRET_KEY
  # aws_session_token session token for temporary security credentials retrieved via STS (optional)
  aws_session_token: $AWS_SESSION_TOKEN
  # assume_role_name is the name of the role to assume (optional)
  assume_role_name: $AWS_ASSUME_ROLE_NAME
  # account_ids is the aws account ids which has similar assumed role name (optional)
  account_ids:
    - $AWS_ACCOUNT_ID_1
    - $AWS_ACCOUNT_ID_2

aws_access_key and aws_secret_key can be generated in the IAM console. We recommend creating a new IAM user with Read Only permissions and providing the access token for the user. Scopes Required: The following scopes can directly be provided to the IAM user.

EC2 - AmazonEC2ReadOnlyAccess
Route53 - AmazonRoute53ReadOnlyAccess
S3 - AmazonS3ReadOnlyAccess
Lambda - AWSLambda_ReadOnlyAccess
ELB - ElasticLoadBalancingReadOnly
Cloudfront - CloudFrontReadOnlyAccess

To also support other services, a custom policy document is provided which can directly be copy-pasted to the role to allow correct and minimal permissions.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "RequiredReadPermissions",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeRegions",
                "route53:ListHostedZones",
                "route53:ListResourceRecordSets",
                "s3:ListAllMyBuckets",
                "lambda:ListFunctions",
                "elasticloadbalancing:DescribeLoadBalancers",
                "elasticloadbalancing:DescribeTargetGroups",
                "elasticloadbalancing:DescribeTargetHealth",
                "cloudfront:ListDistributions",
                "ecs:ListClusters",
                "ecs:ListServices",
                "ecs:ListTasks",
                "ecs:DescribeTasks",
                "ecs:DescribeContainerInstances",
                "eks:ListClusters",
                "eks:DescribeCluster",
                "apigateway:GET",
                "lightsail:GetInstances",
                "lightsail:GetRegions"
            ],
            "Resource": "*"
        }
    ]
} 

References:

Aws Assume Role:
- https://docs.aws.amazon.com/sdkref/latest/guide/feature-assume-role-credentials.html
- https://docs.logrhythm.com/OCbeats/docs/aws-cross-account-access-using-sts-assume-role

GCP (Google Cloud Platform)

Cloudlist supports two approaches for GCP asset discovery:

Organization-Level Asset API - Comprehensive organization-wide discovery using Cloud Asset Inventory API
Individual Service APIs - Fast project-specific discovery using individual GCP service APIs

Individual Service APIs (Default)

Supported GCP Services:

Example Config: Google Cloud Platform can be integrated by using the following configuration block.

- provider: gcp # provider is the name of the provider
  # profile is the name of the provider profile
  id: logs
  # gcp_service_account_key is the minified json of a google cloud service account with list permissions
  gcp_service_account_key: '{xxxxxxxxxxxxx}'

gcp_service_account_key can be retrieved by creating a new service account. To do so, create service account with Read Only access to cloudresourcemanager and dns scopes in IAM. Next, generate a new account key for the Service Account by following steps in Reference 2. This should give you a json which can be pasted in a single line in the gcp_service_account_key. Scopes Required: Cloud DNS, GKE

Organization-Level Asset API

The GCP Asset API provides comprehensive organization-wide asset discovery using the Cloud Asset Inventory API. This approach is more efficient for large organizations with multiple projects and provides a unified view of all assets. Example Config:

- provider: gcp # provider is the name of the provider
  id: organization
  # gcp_service_account_key is the minified json of a google cloud service account with asset inventory permissions
  gcp_service_account_key: '{xxxxxxxxxxxxx}'
  # use_asset_inventory enables the Cloud Asset Inventory API
  use_asset_inventory: true
  # organization_id is the organization ID (e.g., "123456789012")
  organization_id: $GCP_ORGANIZATION_ID
  # project_ids is a list of project IDs to scan (optional - if not provided, all projects in the organization will be scanned)
  project_ids:
    - $GCP_PROJECT_ID_1
    - $GCP_PROJECT_ID_2

Required Permissions for Asset API: For organization-level asset discovery, the service account needs the following IAM roles:

roles/cloudasset.viewer - To read asset inventory data
roles/resourcemanager.organizationViewer - To list organization resources
roles/resourcemanager.folderViewer - To list folder resources (if using folders)
roles/browser - To browse project resources

Custom IAM Policy for Asset API:

{
  "Version": "1",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "cloudasset.assets.searchAllResources",
        "cloudasset.assets.searchAllIamPolicies",
        "cloudasset.assets.listAssets",
        "resourcemanager.organizations.get",
        "resourcemanager.projects.list",
        "resourcemanager.folders.list"
      ],
      "Resource": "*"
    }
  ]
}

Benefits of Asset API:

Comprehensive Coverage: Discovers all supported GCP resource types in one API call
Organization-wide Visibility: Scans across all projects and folders in an organization
Efficient: Reduces the number of API calls compared to individual service APIs
Consistent: Provides a unified view of assets across all GCP services
Historical Data: Access to asset history and change tracking

Supported Asset Types via Asset API: The Asset API can discover a wide range of GCP resources including:

Compute instances, disks, networks, firewalls
Cloud Storage buckets and objects
Cloud SQL instances and databases
Kubernetes clusters and nodes
Cloud Functions
App Engine services
Load balancers and networking resources
And many more…

References:

Azure

Supported Azure Services:

Virtual Machines
Public IP Addresses
Traffic Manager Profiles
Storage Accounts
Azure Kubernetes Service (AKS)
Content Delivery Network (CDN)
DNS Zones and Records
Application Gateway & Load Balancer
Container Instances
App Service & Web Apps
Azure Functions
API Management
Front Door
Container Apps
Static Web Apps

Example Configurations: Microsoft Azure integration supports multiple authentication methods via the modern Track 2 SDK. Here are examples for different scenarios: 1. Client Secret Authentication (Most Common):

- provider: azure
  id: production
  client_id: $AZURE_CLIENT_ID
  client_secret: $AZURE_CLIENT_SECRET
  tenant_id: $AZURE_TENANT_ID
  subscription_id: $AZURE_SUBSCRIPTION_ID

2. Azure CLI Authentication (Local Development):

- provider: azure
  id: development
  subscription_id: $AZURE_SUBSCRIPTION_ID
  use_cli_auth: true

3. Client Certificate Authentication (Enterprise Security):

- provider: azure
  id: secure-env
  client_id: $AZURE_CLIENT_ID
  certificate_path: /path/to/certificate.pem
  tenant_id: $AZURE_TENANT_ID
  subscription_id: $AZURE_SUBSCRIPTION_ID

4. Managed Identity (Azure VMs, App Services, AKS):

- provider: azure
  id: azure-hosted
  subscription_id: $AZURE_SUBSCRIPTION_ID
  use_managed_identity: true

5. Workload Identity (Kubernetes, GitHub Actions OIDC):

- provider: azure
  id: k8s-workload
  subscription_id: $AZURE_SUBSCRIPTION_ID
  use_workload_identity: true

6. Auto-Detection (DefaultAzureCredential):

- provider: azure
  id: auto-detect
  subscription_id: $AZURE_SUBSCRIPTION_ID
  # No explicit auth method - automatically detects available credentials

Authentication Setup:

tenant_id, client_id, client_secret can be obtained/generated from All services > Azure Active Directory > App registrations
subscription_id can be retrieved from All services > Subscriptions
For CLI auth: set use_cli_auth: true and run az login in the terminal
For certificate auth: generate a certificate and register it with your App Registration
For managed/workload identity: ensure the identity has appropriate permissions on the subscription

Authentication Priority (when using auto-detection):

Environment Variables (AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_TENANT_ID)
Workload Identity (Kubernetes service account tokens)
Managed Identity (Azure VMs, App Services, AKS nodes)
Azure CLI (az login session)

References:

Alibaba Cloud

Suppoted Alibaba Cloud Services:

ECS Instances

Example Config: Alibaba Cloud can be integrated by using the following configuration block.

- provider: alibaba # provider is the name of the provider
  # id is the name defined by user for filtering (optional)
  id: staging
  # alibaba_region_id is the region id of the resources
  alibaba_region_id: $ALIBABA_REGION_ID
  # alibaba_access_key is the access key ID for alibaba cloud account
  alibaba_access_key: $ALIBABA_ACCESS_KEY
  # alibaba_access_key_secret is the secret access key for alibaba cloud account
  alibaba_access_key_secret: $ALIBABA_ACCESS_KEY_SECRET

Alibaba Cloud Access Key ID and Secret can be created by visiting https://ram.console.aliyun.com/manage/ak References:

VPS & PaaS Hosting Providers

DO (DigitalOcean)

Supported DigitalOcean Services:

Instances

Example Config: Digitalocean can be integrated by using the following configuration block.

- provider: do # provider is the name of the provider
  # id is the name defined by user for filtering (optional)
  id: xxxx
  # digitalocean_token is the API key for digitalocean cloud platform
  digitalocean_token: $DIGITALOCEAN_TOKEN

digitalocean_token can be generated from the Digitalocean Control Panel. We recommend only giving Read Access to the token. References:

https://www.digitalocean.com/docs/apis-clis/api/create-personal-access-token/

SCW (Scaleway)

Supported Scaleway Services:

Instances

Example Config: Scaleway can be integrated by using the following configuration block.

- provider: scw # provider is the name of the provider
  # scaleway_access_key is the access key for scaleway API
  scaleway_access_key: $SCALEWAY_ACCESS_KEY
  # scaleway_access_token is the access token for scaleway API
  scaleway_access_token: $SCALEWAY_ACCESS_TOKEN

scaleway_access_key and scaleway_access_token can be generated from the Credentials Options in scaleway console. References -

https://www.scaleway.com/en/docs/generate-api-keys/

Heroku

Supported Heroku Services:

Applications

Example Config: Heroku can be integrated by using the following configuration block.

- provider: heroku # provider is the name of the provider
  # id is the name defined by user for filtering (optional)
  id: staging
  # heroku_api_token is the api key for Heroku account
  heroku_api_token: $HEROKU_API_TOKEN

heroku_api_token can be generated from https://dashboard.heroku.com/account/applications/authorizations/new It can also be created with the Heroku CLI by running:

$ heroku authorizations:create -d "brief description of token"
Creating OAuth Authorization... done
Client:      <none>
ID:          a6e98151-f242-4592-b107-25fbac5ab410
Description: brief description of token
Scope:       global
Token:       cf0e05d9-4eca-4948-a012-b9xxxxxxxxxx
Updated at:  Fri Jun 16 2021 13:26:56 GMT-0700 (PDT) (less than a minute ago)

References:

https://devcenter.heroku.com/articles/platform-api-quickstart#authentication

Linode

Supported Linode Services:

Instances

Example Config: Linode can be integrated by using the following configuration block.

- provider: linode # provider is the name of the provider
  # id is the name defined by user for filtering (optional)
  id: staging
  # linode_personal_access_token is the personal access token for linode account
  linode_personal_access_token: $LINODE_PERSONAL_ACCESS_TOKEN

linode_personal_access_token can be created from https://cloud.linode.com/id/tokens. Minimum scope needed is Read Only for Linodes resource. References:

https://www.linode.com/docs/guides/getting-started-with-the-linode-api/#get-an-access-token

CDN & DNS Management

Fastly

Supported Fastly Services:

Services

Example Config: Fastly can be integrated by using the following configuration block.

- # provider is the name of the provider
 provider: fastly
 # id is the name defined by user for filtering (optional)
 id: staging
 # fastly_api_key is the personal API token for fastly account
 fastly_api_key: $FASTLY_API_KEY

fastly_api_key can be generated from https://manage.fastly.com/account/personal/tokens References -

https://docs.fastly.com/en/guides/using-api-tokens#creating-api-tokens

Namecheap

Supported Namecheap Services:

Domain List

Example Config: Namecheap can be integrated by using the following configuration block.

- provider: namecheap # provider is the name of the provider
  # id is the name defined by user for filtering (optional)
  id: staging
  # namecheap_api_key is the api key for namecheap account
  namecheap_api_key: $NAMECHEAP_API_KEY
  # namecheap_user_name is the username of the namecheap account
  namecheap_user_name: $NAMECHEAP_USER_NAME

Namecheap API Access can be enabled by visiting https://ap.www.namecheap.com/settings/opensource/apiaccess/ and then:

Toggle ON API Access switch
Add your public IP to Whitelistted IPs

References:

https://www.namecheap.com/support/api/intro/
- Enabling API Access
- Whitelisting IP

Cloudflare

Supported Cloudflare Services:

Example Config: Cloudflare can be integrated by using the following configuration block.

- provider: cloudflare # provider is the name of the provider
  # email is the email for cloudflare
  email: $CF_EMAIL
  # api_key is the api_key for cloudflare
  api_key: $CF_API_KEY
  # api_token is the scoped_api_token for cloudflare (optional)
  api_token: $CF_API_TOKEN

api_key can be generated from Cloudflare API Key manager. It needs to be Global API Key due to limitation of cloudflare new API tokens. References:

https://developers.cloudflare.com/api/keys

Hetzner Cloud

Supported Hetzner Cloud Services:

Instances

Example Config: Hetzner Cloud can be integrated by using the following configuration block.

- provider: hetzner # provider is the name of the provider
  # id is the name defined by user for filtering (optional)
  id: staging
  # auth_token is the is the hetzner authentication token
  auth_token: $HETZNER_AUTH_TOKEN

References:

https://docs.hetzner.cloud/#authentication

Infrastructure Automation & Management

Kubernetes (K8s)

Supported Kubernetes Services:

Services
Ingresses

Example Config: To integrate Kubernetes, utilize the configuration block outlined below. This block allows you to specify Kubernetes connection details either through a file path or directly by providing the encoded kubeconfig content. If both kubeconfig_file and kubeconfig_encoded are specified, the kubeconfig_encoded will take precedence.

- provider: kubernetes # provider is the name of the provider
  # id is the name of the provider id
  id: staging
  # kubeconfig_file is the path of kubeconfig file
  kubeconfig: path/to/kubeconfig
  # context is the context to be used from kubeconfig file
  context: <context-name>

References:

Hashicorp Tools

With Cloudlist you can get assets from nomad, consul and terraform. Cloudlist can query Nomad and Consul directly, and will use the Terraform state file to list Terraform created assets.

Nomad

Hashicorp Nomad can be integrated by using the following configuration block. Example Config:

- provider: nomad # provider is the name of the provider
  # nomad_url is the url for nomad server
  nomad_url: http:/127.0.0.1:4646/
  # nomad_ca_file is the path to nomad CA file
  # nomad_ca_file: <path-to-ca-file>.pem
  # nomad_cert_file is the path to nomad Certificate file
  # nomad_cert_file: <path-to-cert-file>.pem
  # nomad_key_file is the path to nomad Certificate Key file
  # nomad_key_file: <path-to-key-file>.pem
  # nomad_token is the nomad authentication token
  # nomad_token: <nomad-token>
  # nomad_http_auth is the nomad http auth value
  # nomad_http_auth: <nomad-http-auth-value>

Specifying https in the nomad_url automatically turns SSL to on. All the fields are optional except the nomad_url. References:

https://www.nomadproject.io/api-docs

Consul

Hashicorp Consul can be integrated by using the following configuration block. Example Config:

- provider: consul # provider is the name of the provider
  # consul_url is the url for consul server
  consul_url: http://localhost:8500/
  # consul_ca_file is the path to consul CA file
  # consul_ca_file: <path-to-ca-file>.pem
  # consul_cert_file is the path to consul Certificate file
  # consul_cert_file: <path-to-cert-file>.pem
  # consul_key_file is the path to consul Certificate Key file
  # consul_key_file: <path-to-key-file>.pem
  # consul_http_token is the consul authentication token
  # consul_http_token: <consul-token>
  # consul_http_auth is the consul http auth value
  # consul_http_auth: <consul-http-auth-value>

Specifying https in the consul_url automatically turns SSL to on. All the fields are optional except the consul_url. References:

https://www.consul.io/api-docs

Terraform

Example Config: Terraform can be integrated by using the following configuration block.

- provider: terraform # provider is the name of the provider
  # id is the name defined by user for filtering (optional)
  id: staging
  #tf_state_file is the location of terraform state file (terraform.tfsate) 
  tf_state_file: path/to/terraform.tfstate

OpenStack

Supported OpenStack Services:

Instances

Example Config:

- provider: openstack # provider is the name of the provider
 # id is the name of the provider id
 id: staging
 # identity_endpoint is OpenStack identity endpoint used to authenticate
 identity_endpoint: $OS_IDENTITY_ENDPOINT
 # domain_name is OpenStack domain name used to authenticate
 domain_name: $OS_DOMAIN_NAME
 # tenant_name is OpenStack project name
 tenant_name: $OS_TENANT_NAME
 # username is OpenStack username used to authenticate
 username: $OS_USERNAME
 # password is OpenStack password used to authenticate
 password: $OS_PASSWORD

Open Source

Documentation Index

​Major Cloud Providers

​AWS (Amazon Web Services)

​GCP (Google Cloud Platform)

​Individual Service APIs (Default)

​Organization-Level Asset API

​Azure

​Alibaba Cloud

​VPS & PaaS Hosting Providers

​DO (DigitalOcean)

​SCW (Scaleway)

​Heroku

​Linode

​CDN & DNS Management

​Fastly

​Namecheap

​Cloudflare

​Hetzner Cloud

​Infrastructure Automation & Management

​Kubernetes (K8s)

​Hashicorp Tools

​Nomad

​Consul

​Terraform

​OpenStack

Major Cloud Providers

AWS (Amazon Web Services)

GCP (Google Cloud Platform)

Individual Service APIs (Default)

Organization-Level Asset API

Azure

Alibaba Cloud

VPS & PaaS Hosting Providers

DO (DigitalOcean)

SCW (Scaleway)

Heroku

Linode

CDN & DNS Management

Fastly

Namecheap

Cloudflare

Hetzner Cloud

Infrastructure Automation & Management

Kubernetes (K8s)

Hashicorp Tools

Nomad

Consul

Terraform

OpenStack