Skip to main content

GPU Overview

Part of the Scheduler feature area (axap-gpu-volcano-ui, MFE id 'scheduler'). Sibling docs: gpu-inventory, workload, timetable.

Overview

GPU Overview (schedulerGpuOverview) is the landing page of the Scheduler MFE — a cluster-wide GPU/queue dashboard with a “GPU View / GPU Pool View / Observability” toggle, a KPI row, a GPU topology canvas, a Queue shelf, and priority-run actions. It lives at modules/volcano/dashboard/pages/GpuPoolOverview/index.vue (system mode) / GpuPoolOverviewProject/index.vue (project mode), and drills down into GPU Pool Detail (kanban board) and Workload Log (log viewer).

The backend is Volcano (Kubernetes GPU scheduler) via a FastAPI service (axap-gpu-volcano, prefix /scheduler/v1), plus two auxiliary backends: zmp-core-api (/core/v1, static GPU/node/pool inventory) and zcp-mcm-go (/mcm/resource/v1beta1, DCGM telemetry).

A demo mode is baked into this codebase and is ON by default (modules/volcano/dashboard/demo/isDemo.ts) — disabled only per-tab via ?demo=0. In demo mode this page’s data comes from src/modules/volcano/dashboard/demo/ instead of the real backend.

Route Design

Route constants: constants/routeNames/index.ts (ROUTE_NAMES.X = 'scheduler' + RAW_ROUTE_NAMES.X), paths: constants/routes/index.ts, mapped to lazy components in pages/index.ts. Both a system and a project route table are built from the same buildRoutes() function in router/index.ts; route names are identical between the two so in-page router.push({name}) works regardless of mount mode.

Route nameRaw pathSystem pathProject pathComponent
ROUTE_NAMES.GPU_OVERVIEW (schedulerGpuOverview)/gpu-overview/system/scheduler/gpu-overview/proj/:projectId/scheduler/gpu-overviewGpuPoolOverview/index.vue (system) / GpuPoolOverviewProject/index.vue (project) — pages/index.ts, router/index.ts
ROUTE_NAMES.GPU_POOL_OVERVIEW (alt view, same page)/gpu-pool-overview/system/scheduler/gpu-pool-overview/proj/:projectId/scheduler/gpu-pool-overviewsame component as above (overviewComponent) — router/index.ts
ROUTE_NAMES.GPU_POOL_DETAIL (drill-down, hidden from menu)/gpu-pools/:poolName/system/scheduler/gpu-pools/:poolName/proj/:projectId/scheduler/gpu-pools/:poolNameGpuPoolDetail/index.vue — router/index.ts
ROUTE_NAMES.WORKLOAD_LOG (drill-down, hidden)/gpu-pools/:poolName/workloads/:workloadName/logs/system/scheduler/gpu-pools/:poolName/workloads/:workloadName/logsproject equivalentWorkloadLog/index.vue — router/index.ts

Route registration mechanics (router/index.ts):

  • buildRoutes(paths, opts) produces both the system tree (buildRoutes(ROUTE_PATHS)) and the project tree (buildRoutes(PROJECT_ROUTE_PATHS, {$([regex]::Match({ includeSettings: false, requiresPermission: false, isProject: true }, "(?<=\{)(.*?)(?=\})").Value}))
  • DEFAULT (/) redirects to ROUTE_NAMES.GPU_OVERVIEW — “첫 진입은 GPU Overview” comment confirms this page is the landing page.
  • GpuPoolDetail and WorkloadLog use meta.layout = ROUTE_NAMES.GPU_POOL_OVERVIEW (shared layout wiring with the overview page).
  • Which route table is active is decided at mount time by mfeBootstrap.ts (not by Vue Router itself): it regex-matches window.location.pathname against /proj/{projectId}/scheduler and picks projectRouterRoutes vs. utils.generateRouterRoutes(MFE_ID, routerRoutes) — mfeBootstrap.ts.
  • Project mode is also independently detectable at runtime inside the page via getHostProjectId(), which regex-matches the current pathname for /proj/{id}/ — utils/hostContext.ts. The page uses this (not route meta) to lock the Project scope filter and to scope GPU pools to “my project”.

UI Structure

GPU Overview → GpuPoolOverview/index.vue (or GpuPoolOverviewProject in /proj mode)
├── Header: breadcrumb, title, live clock, live/paused indicator + polling interval select,
│ manual refresh, 3-way view toggle (GPU View / GPU Pool View / Observability)
├── ScopeFilter (Project / Cluster only)
├── AdvancedOverview.vue (viewMode='gpu', default) — KPI row + GpuTopology canvas + NeedsAttentionPanel rail
├── NebulaObservability.vue (viewMode='gpu' && advView='obs') — Shadow-DOM embedded self-contained dashboard
├── QueueShelf (viewMode!=='gpu') — Reserving workload tokens → PriorityRunDialog
├── PoolChassis / ClusterGpuChassis (viewMode='pool') — chassis rows grouped by POOL_CATEGORY_ORDER (FULL/MIG/SLICE)
│ └── GPU Pool Detail (hidden route) → GpuPoolDetail/index.vue (kanban board: queue/running/reserving/completed)
│ └── Workload Log (hidden route) → WorkloadLog/index.vue (split layout: detail rail + log viewer)
├── GpuPoolCreateModal + GpuPoolAdminToast (realm-admin only)
├── HoverPopover
└── PriorityRunDialog

Component Design

pages/GpuPoolOverview/index.vue:

  • Header: breadcrumb, title, live clock, polling live/paused indicator + interval <select> (POLLING_OPTIONS = [5,10,15,30,60]), manual refresh button, and a 3-way view toggle (advView: 'gpu' | 'pool' | 'obs') — lines 1-113.
  • ScopeFilter (Project/Cluster only, hide-name + hide-namespace, cluster-all-option=false) — cluster defaults to profileStore.clusters[0] via a watch; changing cluster/project triggers window.location.reload() (full reload, not a soft refetch) — lines 115-138, onChangeClusterFilter/onChangeProjectFilter.
  • AdvancedOverview (modules/volcano/dashboard/advanced/AdvancedOverview.vue) rendered when viewMode==='gpu' && advView!=='obs': KPI cards (KpiCard), GpuTopology canvas (drag/drop-capable — onDropWorkload, onOpenPool, onOpenQueue→goDetail, onOpenInventory→goInventory), and a right rail NeedsAttentionPanel (Waiting/Idle/Running lists) plus modals: PoolBillingModal, IdleOccupiedModal, IdleLossModal, UtilReviewModal.
  • NebulaObservability (./NebulaObservability.vue) rendered when advView==='obs' — a self-contained Shadow-DOM-embedded dashboard (topology animation, live event log, SVG charts) per its own file comment.
  • QueueShelf (Reserving workload tokens; clicking opens PriorityRunDialog) rendered when viewMode!=='gpu'.
  • Pool-view cluster chassis: PoolChassis (per POOL_CATEGORY_ORDER) when viewMode==='pool', or ClusterGpuChassis (device-card layout) otherwise, both fed by effectiveWorkloads / topologyByPoolName.
  • GpuPoolCreateModal + AdminToast are only visible to adminStore.isRealmAdmin.
  • PriorityRunDialog is a shared modal invoked from QueueShelf click and (per its own file) from GpuPoolDetail’s action popover.
  • GpuPoolDetail/index.vue (drill-down) — kanban board with a custom confirm modal, an action popover (Priority Run / Schedule) with a priority picker (PICKER_GROUPS/priority tiers), reached via navigateToDetail from the chassis.
  • WorkloadLog/index.vue (drill-down) — split layout: left “infor_side” rail (Status / Spec / Resource-per-pod / Placement sections) + right log viewer; still enriches data via legacy Yunikorn-shaped calls (fetchApplications, store.yunikornAppsRaw) rather than the flat VolcanoWorkload model (see Key Findings).

API Design

All calls use services.axiosInstance from @cloudz-mp/zmp-base-ui — evidenced in every apis file, e.g. modules/volcano/dashboard/apis/index.ts#L1-L2.

Base path resolution (modules/volcano/dashboard/apis/index.ts)

  • API_BASE = '/scheduler/v1', CORE_API_BASE = '/core/v1'.
  • resolveRealm() — precedence: URL ?realm= → profileStore.profile.realm → window.RUNTIME_CONFIG.DEFAULT_REALM → VITE_DEFAULT_REALM → "public".
  • resolveCluster() — precedence: URL ?cluster= → profileStore.selectedCluster → profileStore.clusters[0] → runtime config → VITE_DEFAULT_CLUSTER → hardcoded fallback "zcp-ai-cp-eks".
  • resolveOverviewCluster() — GPU Overview only: no explicit selection → "all" (server-side fan-out across clusters), used because resolveCluster()’s single-cluster default is unsuitable for a cluster-wide dashboard.
  • getRealmClusterPath() → ${API_BASE}/realms/${realm}/clusters/${cluster}.
  • getRealmPath() → ${CORE_API_BASE}/realms/${realm} (core-api).
  • Dev-time proxy rewrites: vite.config.ts — /api/scheduler and /scheduler/v1 → axap-gpu-volcano (localhost:8000); /api/core and /core/v1 → zmp-core-api (localhost:8081); /api/mcm and /mcm/resource → zcp-mcm-go (localhost:8080).

Endpoints used by GPU Overview

FunctionMethod & pathNotes
fetchVolcanoQueues()GET {realmCluster}/volcano-queuesprimary queue/pool source (useGpuPoolOverview)
fetchWorkloads(namespace='')GET {realmCluster}/workloadscluster-wide PodGroup scan (no namespace filter — see Business Logic)
elevateWorkloadPriority(namespace, podName, priority)POST {realmCluster}/actions/direct-run?namespace&pod_name&priorityPriorityRunDialog → “Priority Run” action
setPoolState(poolName, state)POST {API_BASE}/clusters/{cluster}/gpu-pools/{name}/stateVolcano Queue Open/Closed toggle (admin)
fetchReservingWorkloads(queueName)GET {realmCluster}/volcano-queues/{name}/reserving-workloadsreserving-workload summary
fetchQueueApplicationDetails(queueName)GET {realmCluster}/applications/queue-detail?queue_nameexplicitly not called from Overview — see Business Logic rule 7
fetchCoreGpuPools() (apis/coreGpu.ts)GET {realmPath}/gpu/pools?size=200derives the Project filter’s option list (clusterProjects)
fetchNodeCapacities(opts?) (apis/coreGpu.ts)GET {realmPath}/gpu/capacities?cluster=via useClusterTopology, pool-view topology
fetchAxapGpuNodes(queue?) (apis/topology.ts)GET /scheduler/v1/clusters/{cluster}/gpu-nodes?queuefallback topology source when core-api/mcm-go unavailable

Response normalization: normalizeWorkload(raw) in apis/index.ts handles two possible backend response shapes — the new flat snake_case VolcanoWorkload schema, and a legacy Kueue-style {metadata, spec, status} shape — converting both into a single VolcanoWorkload client shape, deriving status from PodGroup conditions (Finished→Completed/Failed, Admitted→Running, else Reserving), plus stuck-detection fields (is_stuck, pod_diagnostics, unschedulable_reason/message) when supplied.

Error handling: no structured error normalizer exists for fetchWorkloads/ elevateWorkloadPriority — callers catch and surface raw messages ad hoc. GPU Pool CRUD (create/update/delete, invoked from GpuPoolCreateModal) does have a normalizer (normalizeGpuPoolError in modules/volcano/dashboard/apis/volcanoGpuPool.ts) mapping HTTP 400/409/403/404/501 to typed error kinds.

State Management

  • useQueueStore (modules/volcano/dashboard/stores/createQueueStore.ts, exported as useQueueStore/useSystemQueueStore — modules/volcano/dashboard/stores/index.ts): holds workloads, queueGpuCapacity, queueManagedResources, queueVgpuQuotaPct, queueVgpuMemoryMiB, queueMigBreakdown, queuePoolMeta, isLoading. Populated by store.setVolcanoQueues(qs) / store.setVolcanoWorkloads(wls) inside useGpuPoolOverview’s loadAll(), consumed via storeToRefs.
  • useGpuPoolAdminStore (modules/volcano/dashboard/stores/useGpuPoolAdminStore.ts) — saving/deleting flags, fieldErrors, lastConflictAt (409 timestamp), toast, isRealmAdmin (computed from profileStore.profile.roles — any role named global-administrator, realm-admin, realm-administrator, or ending in admin), currentProject. Gates the “Create GPU Pool” modal visibility.
  • useProfileStore (stores/useProfileStore.ts) — cross-MFE scope store, persisted to localStorage (axap.gpu.selectedCluster/selectedProject/selectedNamespace). GPU Overview reads clusters/projects for its ScopeFilter options and writes selectedCluster/ selectedProject on filter change.
  • Page-local state (not centralized): filterProject, filterCluster, advView, advPoolView, showCreate, clusterProjects, priorityDlg — all plain refs in GpuPoolOverview/index.vue.
  • useClusterTopology() is a plain composable (not a Pinia store) — each call site (this page, and Time Table) gets an independent instance with its own polling timers (topology every 30s, telemetry every 10s).

Business Logic

  1. Workload status derivation — Finished(reason=Failed)→Failed, Finished(other)→ Completed, Admitted→Running, Evicted→Failed, else (QuotaReserved/no condition)→Reserving — modules/volcano/dashboard/stores/createQueueStore.tsgetWorkloadStatus(), re-implemented for the flat shape in mapVolcanoWorkload()/ normalizeWorkload().
  2. Stuck-workload reclassification — if the backend flags is_stuck === true (PodGroup phase says Running but no actual Pod is up), the FE forces status to Reserving so “running” counts never include infra-broken workloads — mapVolcanoWorkload().
  3. Gang scheduling expansion — a workload with min_member > 1 is a gang; expandWorkload() explodes 1 PodGroup into N rows (one per active_pods[] entry).
  4. Priority class ↔︎ numeric value mapping — priorityClassToValue(): p1..p9 → 100..900, default-batch → 100, else null. elevateWorkloadPriority’s priority param and the Priority Run dialog’s tier picker both use this 100-per-tier scheme.
  5. Default/unnamed queues are never surfaced — “정책상 모든 워크로드가 명시적으로 큐를 지정해야 함” — useGpuPoolOverview’s backendPools mapping filters out q.shortName.endsWith('-default').
  6. Pool category heuristic (categorizePool() in modules/volcano/dashboard/constants/poolCategories.ts): explicit POOL_CATEGORY_OVERRIDE wins; else regex on name + gpuProfile — SLICE pattern checked first, then MIG, else default FULL. Drives POOL_CATEGORY_ORDER grouping in PoolChassis/ClusterGpuChassis.
  7. Managed-resource filtering guard — filterByManagedResources() in useGpuPoolOverview/index.ts: a workload whose queueName/poolName explicitly matches the pool is trusted unconditionally (no resource-key check), since “큐 admit 자체가 quota 를 통과한다는 보증”; only ambiguous matches are filtered by queueManagedResources. This is also why queue-detail fetch is intentionally skipped on this page (comment: emptying-queue side effect would hide Pending PodGroups from other queues).
  8. Overview cluster resolution defaults to “all” — resolveOverviewCluster(), because resolveCluster()’s single-cluster default could pick an empty control-plane cluster.
  9. Cluster/Project filter change ⇒ full page reload — onChangeClusterFilter/ onChangeProjectFilter call window.location.reload() rather than an in-place refetch.
  10. Project mode locks scope — when getHostProjectId() is non-null, the Project scope filter is locked and forced to the host project id; GPU Overview additionally renders a separate GpuPoolOverviewProject component restricting to project-owned pools (exact internal narrowing logic not traced — see Assumptions).
  11. Demo mode is default-ON and cannot be persisted OFF — isDemoMode() always returns true unless the current URL has ?demo=0 (ignores any previous localStorage lock).

Data Flow

flowchart TD
R1[/system/scheduler/gpu-overview/] --> P1[GpuPoolOverview/index.vue]
P1 --> H1[useGpuPoolOverview]
P1 --> H2[useClusterTopology]
H1 --> S1[useQueueStore]
H1 --> A1[apis: fetchVolcanoQueues / fetchWorkloads]
H1 --> A2[apis/coreGpu: fetchCoreGpuPools]
H2 --> A3[apis/coreGpu: fetchNodeCapacities]
H2 --> A4[apis/topology: fetchAxapGpuNodes]
P1 -->|Priority Run click| A5[apis: elevateWorkloadPriority]
A1 & A2 & A3 & A4 & A5 --> BE[(axap-gpu-volcano /scheduler/v1\ncore-api /core/v1)]
P1 -->|navigateToDetail| R2[GpuPoolDetail/index.vue]
R2 --> R3[WorkloadLog/index.vue]

loadAll() (inside useGpuPoolOverview) is the single source of truth for this page’s queue/workload data; useClusterTopology() is a separate, independently-polling data source for GPU topology used only in viewMode==='pool'.

Dependency Graph

flowchart LR
GpuPoolOverview --> useGpuPoolOverview
GpuPoolOverview --> useClusterTopology
GpuPoolOverview --> useGpuPoolAdminStore
GpuPoolOverview --> useProfileStore
GpuPoolOverview --> apisIndex["apis/index.ts (scheduler/v1)"]
GpuPoolOverview --> apisCoreGpu["apis/coreGpu.ts (core/v1)"]
useGpuPoolOverview --> useQueueStore
useGpuPoolOverview --> apisIndex
useClusterTopology --> apisCoreGpu
useClusterTopology --> apisMcm["apis/mcm.ts"]
useClusterTopology --> apisTopology["apis/topology.ts"]
GpuPoolDetail --> useQueueStore
WorkloadLog --> useQueueStore
WorkloadLog --> apisIndex
useGpuPoolAdminStore --> apisVolcanoGpuPool["apis/volcanoGpuPool.ts"]
apisIndex --> useProfileStore
apisCoreGpu --> apisIndex

Sequence Flow

A. GPU Overview initial load

sequenceDiagram
actor User
participant Page as GpuPoolOverview/index.vue
participant Hook as useGpuPoolOverview
participant Store as useQueueStore
participant API as apis/index.ts
participant BE as axap-gpu-volcano

User->>Page: navigate to /system/scheduler/gpu-overview
Page->>Hook: useGpuPoolOverview()
Hook->>API: fetchVolcanoQueues()
API->>BE: GET /scheduler/v1/realms/{realm}/clusters/{cluster}/volcano-queues
BE-->>API: VolcanoQueueResponse[]
API-->>Hook: queues
Hook->>Store: setVolcanoQueues(qs)
Hook->>API: fetchWorkloads('')
API->>BE: GET .../workloads
BE-->>API: raw items (2 possible shapes)
API->>API: normalizeWorkload() per item
API-->>Hook: VolcanoWorkload[]
Hook->>Store: setVolcanoWorkloads(wls)
Store-->>Hook: workloads (reactive, via storeToRefs)
Hook-->>Page: allPools / workloads / isLoading / lastUpdated
Page->>Page: render PoolChassis/ClusterGpuChassis/AdvancedOverview

B. Priority Run action (User Action → … → UI Update)

sequenceDiagram
actor User
participant Shelf as QueueShelf
participant Dialog as PriorityRunDialog
participant API as apis/index.ts
participant BE as axap-gpu-volcano
participant Page as GpuPoolOverview/index.vue

User->>Shelf: click a Reserving workload token
Shelf->>Page: emit click-workload
Page->>Dialog: openPriorityRun() sets priorityDlg.open=true
User->>Dialog: pick priority tier (P1..P9) and Confirm
Dialog->>Page: emit pick(priority)
Page->>API: elevateWorkloadPriority(namespace, podName, priority)
API->>BE: `POST /scheduler/v1/realms/{r}`/clusters/{c}/actions/direct-run?namespace&pod_name&priority
BE-->>API: {success, message, detail?}
API-->>Page: response
Page->>Page: closePriorityRun(); refresh() (re-fetch queues/workloads)
Page-->>User: dialog closes, queue shelf / block colors update on next reload

Key Findings

  • Two parallel data models coexist for “workload”: a modern flat Volcano VolcanoWorkload (apis/index.ts normalizeWorkload) and a legacy Yunikorn-flavored ApplicationDetail model still referenced by useQueueStore and WorkloadLog/index.vue (fetchApplications, appRaw, stateLog) — an in-progress Yunikorn→Volcano migration.
  • “GPU Pool View” and “Observability” are largely demo/mock-backed: AdvancedOverview.vue imports mockAdvanced.ts; NebulaObservability runs a self-driven simulation (“자체 sim 으로 동작” per its own comment), independent of any live backend call.
  • Demo mode defaults to ON app-wide, meaning a fresh clone of this repo, run without ?demo=0, renders synthetic data even when a real backend is reachable.
  • Repo-root CLAUDE.md/AGENTS.md describe a different module path (src/modules/ scheduler/dashboard/, MFE id mfe-workload) than the actual code (src/modules/volcano/dashboard/, MFE_ID = 'scheduler' — vite.config.ts). This document trusts only the verified code paths cited above.

Risks / Technical Debt

  • console.log calls left in production API code (fetchVolcanoQueues) log request URLs/responses unconditionally.
  • Two independent status-derivation implementations (getWorkloadStatus() vs. inline logic in mapVolcanoWorkload()/normalizeWorkload()) risk drifting out of sync.
  • No shared error normalizer for workload/queue APIs (only GPU Pool CRUD has one).
  • Full-page window.location.reload() on cluster/project filter change discards all in-memory state and re-runs the entire mount/profile-fetch sequence.
  • Hardcoded cluster fallback "zcp-ai-cp-eks" in resolveCluster() is an environment-specific value baked into shared code.
  • GpuPoolOverviewProject/index.vue was not deeply inspected — confirmed to exist and be wired into the router/pages map, but its internal “my project pools” narrowing logic is Not Found in Code in this document (not traced line-by-line).

Assumptions

  • Backend services (axap-gpu-volcano, zmp-core-api) are external repos; their actual endpoint implementations are Not Found in Code in this workspace — inferred only from FE request paths/comments and response-shape TypeScript interfaces.
  • STORES constant (Pinia store IDs) was referenced but its enumeration file (constants/stores/index.ts) was not opened; exact store ID strings are Not Found in Code in this document.
  • Unit/e2e test coverage for this page was Not Found in Code beyond an apis/tests/ directory listing (contents not inspected).