Cloud Billing Monitor (GCP)

Result

• Result Translation
• GCP aichip 비용 보고서 (2024-11-24) == GCP project name aichip’s costs daily Reports (2024-11-24)
• 이번달 누적 금액 == This month total costs
• 전일 == Yesterday
• SKU 전일 사용 금액 (1000원 이상만 표시) ( Costs of each SKU , which is Over 1$)
 
 
 

Required IAM

• Billing Account Administrator role

• BigQuery User role 

How to do?

1. Set Billing Export at Billing Section

2. Set BigQuery dataset

3. Use Slack Webhook

• Copy your slack webhook url

4. Make Serviceaccount and download into json to use python code with slack webhook

Sample Codes

```json
{
  "type": "service_account",
  "project_id": "[your project id],
```

# setup slack webhook url

```python
SERVICE_ACCOUNT_PATH = 'project id.json'
SLACK_WEBHOOK_URL = "https://hooks.slack.com/services/...[YOUR SLACK URL]"

```

# BigQuery serviceaccount setups and SQL 

```python

def get_gcp_costs_by_sku():
    credentials = service_account.Credentials.from_service_account_file(SERVICE_ACCOUNT_PATH)
    client = bigquery.Client(credentials=credentials, project=PROJECT_ID)
		...

    sku_query = f"""
        SELECT
            sku.description AS sku_description,
            SUM(cost) AS total_cost,
            SUM(usage.amount) AS total_usage,
            ANY_VALUE(usage.unit) AS usage_unit
        FROM
            `{PROJECT_ID}.[YOUR DATASET NAME]`
        WHERE
            usage_start_time >= '{start_of_month}'
            AND usage_start_time < '{today}'
        GROUP BY
            sku_description
        HAVING
            total_cost > 100
        ORDER BY
            total_cost DESC
    """

    sku_result = client.query(sku_query).result().to_dataframe()
```

CronJob with k8s

• containerize simply

FROM python:3.12-slim
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt
CMD ["python", "billing.py"]

apiVersion: batch/v1
kind: CronJob
metadata:
  name: mobidoc-billing-report
  namespace: default 
spec:
  **timeZone: 'Asia/Seoul' # only work in k8s 1.27+ 
  schedule: "0 9 * * *"**
  jobTemplate:
    spec:
      template:
        metadata:
          annotations:
            sidecar.istio.io/inject: "false" #istio 깔려있다면 필요에 따라 설정
        spec:
          containers:
          - name: mobidoc-billing
            image: registry.gocap.kr/etc/mobidoc-billing:0.0.2
            imagePullPolicy: Always
            env:
              - name: TZ
                value: "Asia/Seoul" 
          restartPolicy: OnFailure

Key Learnings and Insights

1. Automated Cloud Cost Reporting
• Designed a system to extract, store, and analyze billing data using BigQuery, allowing real-time cost monitoring without relying on the GCP console.
• Gained hands-on experience with SQL-based cost aggregation and filtering strategies to extract meaningful insights.

2. Kubernetes CronJobs for Scheduled Automation
• Implemented Kubernetes CronJobs to automate daily billing reports, reducing manual monitoring efforts.
• Explored timezone configurations and Istio sidecar management in K8s batch processing environments.

3. Slack Webhook Integration for Real-Time Notifications
• Developed a Slack bot to automatically send cost reports, improving operational visibility.
• Learned best practices for integrating cloud monitoring with collaboration tools, enabling faster response to cost anomalies.