AI Core and GenAI Hub: Architecture and Use

JinJin48

SAP Consultant | BTP · S/4HANA · US Global Rollout

May 9, 2026 · 13 min read

Where This Article Sits#

References#

1) AI Core and GenAI Hub: Architecture and Use#

Separating these names is the first job on any AI Core engagement.

They are not substitutes.

2) How It Connects to S/4HANA#

+------------------------------------------------------------+
|     Application code (CAP, ABAP Cloud, custom Cloud Foundry)|
+----------------------+-------------------------------------+
                       |  SAP Cloud SDK for AI (Java / JS / Py)
+----------------------v-------------------------------------+
|                  SAP BTP (Cloud Foundry / Kyma)            |
|                                                            |
|   +-------------------------+    +----------------------+  |
|   |  SAP AI Core (Extended) |<-->|  Generative AI Hub   |  |
|   |  - Orchestration depl.  |    |  Model catalog       |  |
|   |  - Custom (BYOM) depl.  |    |  Routes via SAP-     |  |
|   |  - Resource groups      |    |  controlled contracts|  |
|   +-----------+-------------+    +----------+-----------+  |
|               |                              |             |
|               v                              v             |
|   +-------------------------+    +----------------------+  |
|   |  Object Store (S3)      |    | Anthropic / OpenAI / |  |
|   |  for grounding RAG      |    | Google / Meta / IBM /|  |
|   +-------------------------+    | Mistral endpoints    |  |
|                                  +----------------------+  |
+------------------------------------------------------------+
                       ^
                       |  Destinations (with IAS principal propagation)
+----------------------+-------------------------------------+
| S/4HANA Cloud / Public Edition / on-prem (read-only OData) |
+------------------------------------------------------------+

Key point: AI Core is a generic AI runtime on BTP, not a feature inside S/4HANA. Your application calls the orchestration deployment URL; S/4HANA data only enters the prompt if your code (or grounding RAG over an S3 doc store) puts it there. Authentication, destination wiring, and PII masking are consultant problems, not platform-team problems.

3) Maturity Map#

4) Requirements Checklist#

Six sub-tables, one per decision area. Walk through these with the customer at requirements stage — before the procurement conversation.

Foundation model selection#

Entitlement and provisioning#

Orchestration configuration#

Data residency, privacy, and compliance#

Cost estimation and metering#

BYOM (only when foundation models do not fit)#

5) Decision Criteria#

The three questions to ask first#

Q1. What concrete, measurable, repeatable task will this LLM do — and how will we measure success?

“We want to use AI in the project” is not an answer. Pick one workflow — invoice-line classification, contract-clause extraction, ABAP refactoring, ticket triage — and define accuracy or human-handoff metrics before model selection.

Q2. What data crosses the prompt boundary, and where can it legally go?

Public product info has very different constraints than employee comp or customer contracts. Identify the data class, then apply residency and masking implications. This question often forces orchestration with masking and grounding from day one.

Q3. Who owns the AI Core CU budget — and have they signed the forecast?

Generative AI consumption is invisible in CPEA dashboards until it isn’t. Forecasts must come from real prompt sizes, not “GPT is cheap” anecdotes. If no one signs the forecast, the project is not yet a project.

Domain selection by project context#

Licensing and cost#

6) Common Failures#

Failure 1: Buying the Standard plan and discovering Generative AI Hub is not in it.

The first project-week setback. Standard is for predictive AI; Generative AI Hub is Extended-only, and the BTP booster does not select Extended automatically. Verify the entitlement screen shows “extended” with non-zero quota before kickoff.

Failure 2: Skipping resource-group provisioning via the AI Core API.

Resource groups are the isolation boundary inside an AI Core instance, and BTP cockpit cannot create them — the AI Core API is the only path. Teams that try to click their way through provisioning end up with deployments stuck in the default group.

Failure 3: Building grounding without owning the document corpus lifecycle.

RAG looks magical in a demo. In production, the failure mode is corpus rot: documents change, owners leave, vectors get stale. Without a refresh SLA and a content owner per corpus, grounding accuracy degrades silently.

Failure 4: Token forecasts based on request count, not token volume.

“100,000 calls per month” says nothing about cost. 300-in / 1500-out = 1.8M tokens per 1000 calls. Forecast in tokens, input plus output, since output costs 2–5× more per unit.

Failure 5: Treating model selection as the architecture decision.

Model is the most-discussed and least-impactful decision. Orchestration design (templating, grounding, masking, filtering, fallback) determines whether the system is operable; model is a tunable parameter inside that design. Teams that pick the model first usually rebuild later.

Failure 6: Forgetting data-masking on one endpoint.

Masking is opt-in per orchestration call. If a developer copies a working config and forgets to enable masking on a new endpoint, raw PII goes to the third-party LLM. This is a compliance finding, not a warning. Add “masking enabled where data class requires it” as an explicit code-review gate.

FAQ#

Q: Can we use the Generative AI Hub from on-premise S/4HANA?

Yes, but the AI logic lives on BTP, not in the on-premise system. Pattern: on-prem S/4HANA exposes OData; a BTP application reads it (principal propagation), calls the orchestration deployment, returns the result. There is no “AI Core for on-premise” planned for 2026.

Q: How is the Generative AI Hub licensed?

Through the AI Core Extended plan, with token usage metered into Capacity Units billed against existing CPEA / cloud credits. AI Launchpad is included. No separate Gen AI subscription. BYOM workloads use predictive-AI compute pricing instead of tokens.

Q: How do we document the LLM selection decision so it survives audit?

Use the AI Launchpad Compare workflow against 3–5 candidate models with at least three representative prompts each. Capture latency, token usage, and qualitative notes. Record the selection in an ADR that references the comparison run, the regional model-availability snapshot, and SAP Note 3437766 for CU rates at decision time.

Q: Are prompts and completions used to train the third-party models?

No. SAP integrates Anthropic, OpenAI, Google, and others under enterprise contracts prohibiting training on customer data. SAP holds ISO/IEC 42001 covering AI Core, Launchpad, and Joule — cite the Trust Center reference when customers ask, which they always do.

Q: How does AI Core relate to Joule?

Joule is the end-user assistant; AI Core is the developer platform underneath. Joule Studio extends Joule with skills that call the Hub. “Chatbot for end users” → Joule first. “Custom AI in our application” → AI Core. Complements, not alternatives.

Summary#

AI Core and Generative AI Hub are mature enough for production in 2026, but project failures cluster around setup, governance, and budget — not model quality. Get the Extended plan and resource group right, design orchestration first and pick the model second, force a token-based budget conversation before procurement. The catalog is generous, the SDK is solid, the audit story is real. What customers need from consultants is the discipline to ask the three questions before reaching for the API.

What to Read Next#