Azure Event Grid and Storage Queue for VM Events

System Topic → Topic Subscription → Storage queue

Duplicated dispatch are controlled by the number of ‘topic subscriptions’

System Topic

We are using ‘system topic’ instead of ‘custom topic’ because the ‘events’ are from Azure, not from our own service.

resource "azurerm_eventgrid_system_topic" "resources" {
  name                = "resource-events-${var.env}"
  resource_group_name = azurerm_resource_group.topics.name
 
  topic_type = "Microsoft.ResourceNotifications.Resources"
  location   = "Global"
 
  source_arm_resource_id = "/subscriptions/${data.azurerm_subscription.current.subscription_id}"
}

What is system topic?

Typic type “Microsoft.ResourceNotifications.Resources”

• captures all resource creation events (including VM, storage account, etc)
• capture all resource udpate events (including configuration changes, scaling, etc)
• captures all resource deletion events (resource termination)
• captures resource state changes (provisioning states, power states, etc)

Locaiton: global:

• all ‘regions’

Source ARM resource ID:

• this basically the ‘scope’ of the events we want to capture

Key Constraint:

We can only hae ONE “Microsoft.ResourceNotifications.Resources” system topic per subscription.

The constraint is on the topic_type = "Microsoft.ResourceNotifications.Resources" string. Not on the type azurerm_eventgrid_system_topic

I.e. you can create multiple azurerm_eventgrid_system_topic resources, but only one of them can have topic_type = "Microsoft.ResourceNotifications.Resources"

Then, there will be other types of constraints:

Topic Type	What It Captures	Limitation
Microsoft.ResourceNotifications.Resources	All resource lifecycle events	1 per subscription
Microsoft.Storage.StorageAccounts	Storage account events	1 per storage account
Microsoft.EventHub.Namespaces	Event Hub events	1 per Event Hub namespace
Microsoft.ServiceBus.Namespaces	Service Bus events	1 per Service Bus namespace
Microsoft.KeyVault.vaults	Key Vault events	1 per Key Vault
Microsoft.ResourceNotifications.HealthResources	Resource health events	1 per subscription
Microsoft.ContainerRegistry.registries	Container Registry events	1 per registry

Microsoft.ResourceNotifications.Resources: this is only available to use with azurerm_eventgrid_system_topic

• It can’t be used with other Azure Event Grid resoruces types like a custom topics.

Event Subscription

Example: Assume we use one ‘topic subscription’ for ‘create or update’ events:

resource "azurerm_eventgrid_system_topic_event_subscription" "vm_createdorupdated_events" {
  name = "vm-createofupdate-events-${var.queue_suffix}"
  system_topic = var.system_topic_name
  resource_group_name = var.resource_group_name
 
  // Send to this storage queue. Note that:
  // - We used the full path of storage account ID.
  // - We used the queue name (not the full path of the queue).
  storage_queue_endpoint {
	storage_account_id = var.queue_storage_account_id
	queue_name = "vm-events-${var.queue_suffix}"
  }
 
  // Filter to determine what events to send to the storage queue
  advanced_filter {
    string_in {
      key    = "Data.resourceInfo.type"
      values = ["Microsoft.Compute/virtualMachines"]  # Only VMs
    }
    string_in {
      key    = "eventType"
      values = ["Microsoft.ResourceNotifications.Resources.CreatedOrUpdated"]  # Creation/Updates
    }
    string_in {
      key    = "Data.resourceInfo.properties.provisioningState"
      values = ["Succeeded"]  # Only successful operations
    }
  }
}

We can create another topic subscription for ‘delete’ event, just change the filter to:

  advanced_filter {
    string_in {
      key    = "Data.resourceInfo.type"
      values = ["Microsoft.Compute/virtualMachines"]
    }
 
    string_in {
      key    = "eventType"
      values = ["Microsoft.ResourceNotifications.Resources.Deleted"]
    }
  }	

NOTE

If we need to have the messages from ‘one’ topic to be dispatched to ‘multiple’ storage queues, we can create multiple topic subscriptions.

Example:

// using the system topic as an example
resource "azurerm_eventgrid_system_topic" "resources" {
  name                = "resource-events-${var.env}"
  topic_type          = "Microsoft.ResourceNotifications.Resources"
  source_arm_resource_id = "/subscriptions/${subscription_id}"
}
 
# Multiple Event Subscriptions (each routes to different queue)
resource "azurerm_eventgrid_system_topic_event_subscription" "vm_events_to_queue_1" {
  name                = "vm-events-queue1"
  system_topic        = azurerm_eventgrid_system_topic.resources.name
  resource_group_name = azurerm_resource_group.topics.name
 
  // Route to queue 1
  storage_queue_endpoint {
    storage_account_id = "/subscriptions/.../storageAccounts/storage1"
    queue_name         = "vm-events-queue1"
  }
 
  advanced_filter {
    string_in {
      key    = "Data.resourceInfo.type"
      values = ["Microsoft.Compute/virtualMachines"]
    }
  }
}
 
resource "azurerm_eventgrid_system_topic_event_subscription" "vm_events_to_queue_2" {
  name                = "vm-events-queue2"
  system_topic        = azurerm_eventgrid_system_topic.resources.name
  resource_group_name = azurerm_resource_group.topics.name
 
  // Route to queue 2
  storage_queue_endpoint {
    storage_account_id = "/subscriptions/.../storageAccounts/storage2"
    queue_name         = "vm-events-queue2"
  }
 
  advanced_filter {
    string_in {
      key    = "Data.resourceInfo.type"
      values = ["Microsoft.Compute/virtualMachines"]
    }
  }
}

┌─────────────────────────────────────────────────────────────────┐
│                    Event Dispatching                            │
├─────────────────────────────────────────────────────────────────┤
│ Azure VM Created                                                │
│           ↓                                                     │
│ System Topic: Microsoft.ResourceNotifications.Resources         │
│           ↓                                                     │
│ Event Grid Routes to ALL Matching Subscriptions:                │
│           ├── Subscription 1 → Storage Queue 1                  │
│           ├── Subscription 2 → Storage Queue 2                  │
│           └── Subscription 3 → Storage Queue 3                  │
│           ↓                                                     │
│ Each Queue Gets the SAME Event                                  │
└─────────────────────────────────────────────────────────────────┘

Storage queue

Storage queue can be provisioned just in one subscription (or two if your services use two subscriptions). But, what I want to say is, this is only to be associated with your own queue consumer service. You don’t need to find a way to ‘automatically’ provision to many ‘subscriptions’ anymore. (i.e. no long multi-tenant)

// Assume we have a resource group specifically for the storage queue related stuffs
resource "azurerm_resource_group" "vmeventsresgrp" {
  name     = "vm-events-${var.env}"
  location = var.region
}
 
// We need a 'storage account' under the current resource group.
// Note that storage account is actually a 'regional' resource.
resource "azurerm_storage_account" "vmeventsacct" {
  name                            = "vm-events-${var.env}"
  resource_group_name             = azurerm_resource_group.vmeventsresgrp.name
  location                        = azurerm_resource_group.vmeventsresgrp.location
  account_tier                    = "Standard"
  account_replication_type        = "GRS"
  allow_nested_items_to_be_public = false
  shared_access_key_enabled       = false
}
 
// Storage queue is a entity belonging to a storage account.
resource "azurerm_storage_queue" "vmeventsqueue" {
	count                = 1
	name                 = "vm-events-${var.env}"
	storage_account_name = azurerm_storage_account.vmeventsacct.name
}

Other

For the topic subscription module, we can route the message to other types of resources, such as:

# Webhook endpoint
webhook_endpoint {
  url = "https://myapp.com/webhook"
}
 
# Azure Function
azure_function_endpoint {
  function_id = "/subscriptions/.../functions/myfunction"
}
 
# Event Hub
eventhub_endpoint {
  eventhub_id = "/subscriptions/.../eventhubs/myhub"
}
 
# Service Bus Queue
service_bus_queue_endpoint {
  queue_id = "/subscriptions/.../queues/myqueue"
}
 
# Service Bus Topic
service_bus_topic_endpoint {
  topic_id = "/subscriptions/.../topics/mytopic"
}

BTW, ‘storage queue’, ‘service bus queue’, ‘service bus topic’, ‘event hub’ are all ‘queue’ type resources on Azure. Let’s copmare them compare-azure-queues

Event Grid = System Topic + Subscriptions + Routing Logic

• The Storage Queue is just the destination where Event Grid delivers the filtered events. Your consumer application polls these queues, but the intelligent event capture, filtering, and routing is all handled by Event Grid.
• So yes, both the “System Topic” and “Topic Subscriptions” are Event Grid components working together to provide the event-driven architecture.