Resilient AI Infrastructure [Bundle of 4 Books]

For those who want to build controlled, reproducible AI systems entirely within your own infrastructure, this book is the most practical and implementation-focused trainer. Instead of relying on external APIs or cloud-hosted intelligence services, this book clearly demonstrates how Apache Spark can orchestrate data preparation, model training, batch inference, reporting, and LLM acceleration in a disciplined and transparent way.

As the book opens, it swiftly defines private AI, making it clear that external AI calls are not allowed, full ownership of datasets and model assets is imperative, and repeatable runs with traceable outputs are essential. I will use a realistic sample to show you how to build an end-to-end workflow that ingests raw data, normalizes it into a stable schema, trains a baseline classifier, extracts keywords, generates summaries, and produces structured reports. There's no doubt that each step is implemented with clarity and attention to maintainability. You can be sure that logging, manifests, and monitoring are embedded from the start. We implement classic machine learning techniques, vLLM, performance measurement, batch processing patterns, quarantine handling, and structured metrics to make private AI more usable and compete with cloud-based AI.

Beyond experimentation, the book transitions seamlessly into packaging and routine execution. It will teach you to bundle multiple stages into a single command workflow, schedule daily or weekly runs, generate compact run reports, and adapt the architecture to new datasets without redesigning the system. It does not promise instant transformation or one-click AI solutions. Instead, it provides a structured path to building a sustainable private AI backbone using Spark as the orchestration layer.

• No external AI calls and full control over data, models, and repeatable runs.
• Stable canonical schema with downstream ML and reusable reporting.
• Infuse Classic ML with Spark without introducing LLM complexity.
• Carry out extractive summaries without hallucination risk.
• Complete traceability through manifests, prompt versions, and run logs.
• Implement data and batch flow, along with fast inference using vLLM.
• Extract inspectable data and surface out the hidden errors using quarantine tables.
• Measure and store performance for every run with stakeholder reporting.
• Design single-command pipeline with clear configs to build repeatable AI.

1. Up and Running with Private AI
2. Data Workflows using Spark DataFrames
3. Powerful NLP without LLM
4. Batch Inference and Practical Outputs
5. Smart Summaries
6. Boosting with vLLM Integration
7. Packaging Private AI

How long do you think it would take you to notice if your AI system started giving worse answers?

Honestly, for most teams, it's weeks, and the news comes from a customer. Chaos engineering answers that question for infrastructure in about thirty seconds, because if a server goes down, it either reroutes or it doesn't. And language models completely shatter all those assumptions. Their output varies for real, their decay builds up instead of happening straight away, and those failures are confident, plausible, and wrong while latency, error rate, and pod health stay exactly where they should be.

This book sets out the new framework for AI infrastructure. You'll find that, working through one support-assistant application across twelve chapters, everything moves from defining steady state for probabilistic systems to injecting faults at every layer, including accelerators, inference servers, model artefacts, retrieval pipelines, model APIs, autonomous agents, distributed training and the monitoring stack itself.

Just to be clear, the toolchain is intentionally kept small. The LitmusChaos orchestrates, the Chaos Mesh reaches the kernel and filesystem, and the mitmproxy handles the content-level faults that neither can touch, extended with custom injectors where nothing suitable exists. So, every experiment comes to one of these four conclusions, and then the last chapter sums them all up as a programme scorecard.

• Write down what "working correctly" means when your system's output legitimately varies.
• Break GPUs and inference servers on purpose, with a restore path that always runs.
• Spot the failures that keep every dashboard green and every alert silent.
• Check model weights for damage before loading them, not after serving starts.
• Catch the model release that scores better overall while getting worse where it matters.
• Find out whether your RAG pipeline is helping, or quietly making answers worse.
• Corrupt what the model returns, then watch whether your agent notices.
• Stop your remediation agent from making an incident worse than it was.
• Prove your training checkpoints actually resume the run they claim to continue.
• Catch slow decay in hours instead of days by changing how you alert.

1. Why AI Systems Fail Differently?
2. Steady State for Probabilistic System
3. Lab and the Toolchain
4. Accelerator and Compute Chaos
5. Inference Serving Chaos
6. Model Artifacts, Registries, and Version Skew
7. Retrieval and Context Chaos
8. Model-Boundary Chaos
9. Agentic and Multi-Agent Chaos
10. Distributed Training and Choas Fine-Tuning
11. Observing Probabilistic Degradation
12. Operationalizing Chaos

Stop switching tabs. Start finding out what's going on.

Whenever there's a major outage, it always starts the very same way. Five tools are open, four teams are talking past each other, and one question nobody can settle quickly. The idea behind this book is to give engineers the tools they need to keep an eye on things without breaking the bottom line.This book brings together logs, metrics, traces, frontend sessions and AI telemetry into one place, so you can figure out what's going on when things go wrong. It's all about setting up a production-ready OpenObserve cluster, hooking up every signal through OpenTelemetry, and tweaking the data until long-term storage becomes budget-friendly.

There are several chapters that look at what can go wrong in a business, like infrastructure, networks, apps, data layers and user experience. They even talk about the AI service that went up by three times overnight. In the AI era, the way we think about failure has changed. Things like models drifting, prompts inflating and cost moving faster than latency. If you've got observability built for three signals, you won't be able to see any of them. This book shows what a unified platform can do.

• Build a signal inventory tying every business driver to a measurable objective.
• Deploy a high-availability OpenObserve cluster with object storage, SSO, and role-based access.
• Instrument any stack through OpenTelemetry without rewriting application code or locking into a vendor.
• Shape telemetry with pipelines, VRL functions, and enrichment tables to cut ingest cost sharply.
• Design dashboards that engineers can actually open during incidents rather than decorative executive wall displays.
• Define burn-rate alerts and SLOs that flag real degradation without generating alert fatigue.
• Apply a repeatable seven-step diagnostic loop from scope through evidence to verified repair.
• Recognise infrastructure red flags like CPU throttling, disk eviction, and autoscaler thrash early.
• Diagnose AI-layer failures including token inflation, prompt drift, and runaway cost per request.
• Migrate from an existing platform using dual-run, honest measurement, and reversible cutover.

1. Getting Started with OpenObserve
2. How Systems Fail with Hidden Failures?
3. OpenObserve and Production-Ready Environment
4. Getting Every Signal In
5. Shaping, Governing and Affording Data
6. Building Dashboards Engineers Actually Use
7. Flagging Engine, Alerts, SLOs and Incidents
8. Diagnostic Method
9. Infrastructure and Kubernetes Red Flags
10. Network and Edge Red Flags
11. Application and Service Red Flags
12. Data-Layer Red Flags
13. User-Facing and AI-Layer Red Flags
14. Scaling OpenObserve

Whether you're an SRE, platform engineer or architect, you'll find configuration, queries, dashboards and alert definitions you can apply directly. You don't need to have experience with OpenObserve.

At some point, every software engineering organisation hits a point where shipping software feels like it's taking longer than writing it. So, developers are waiting days for a database, environments are drifting apart, and every team is deploying in its own way. This book shows you how to do that, one working layer at a time.

In this book, you'll get to build a complete internal developer platform using tools you can run on a single Linux workstation. The plan is to provision multi-cluster infrastructure with OpenTofu, wire GitOps delivery with Argo CD, design a self-service component API with Kubebuilder and Crossplane, enforce policy with Kyverno and Cilium, and add observability and a Backstage portal that cuts onboarding from weeks to minutes. Then you'll adopt OpenChoreo and use it with custom component types, promotion pipelines, multi-cluster data planes, and AI agents connected through MCP.

Out of the book, about 40% is about the thinking behind it, and the rest is all about putting it into practice. The idea is to make you capable and skilled to design, build, secure, measure and grow a platform that developers will actually use.

• Provision multi-cluster Kubernetes foundations with OpenTofu modules, gateways, and shared secret stores.
• Deliver change through GitOps with Argo CD, ApplicationSets, and progressive rollouts.
• Design single-file component API that hides Kubernetes detail from developers.
• Provision databases on demand with Crossplane compositions and a custom operator.
• Enforce security baselines through Kyverno policies, signed images, and zero-trust network rules.
• Instrument services with OpenTelemetry, and reuse dashboards, SLOs, burn-rate alerts.
• Speedup onboarding using Backstage templates, catalog entities, and TechDocs.
• Operate OpenChoreo with custom component types, traits, and promotion pipelines.
• Run dedicated production data planes across clusters using outbound cluster agents.
• Grow adoption voluntarily, score platform maturity, and prove ROI to leadership.

1. Why Platform Engineering Exists
2. Thinking Like a Platform Team
3. Architecting Internal Developer Platform
4. Building Foundation Layer
5. GitOps and Delivery Pipelines
6. Self-Service and Platform APIs
7. Guardrails, Security and Governance
8. Observability and Developer Portals
9. OpenChoreo as Reference Outcome
10. Operating OpenChoreo
11. Scaling Platform Adoption

If you know to ship software, and have worked around Git, containers and the Linux command line, then this book is well-suited for you. No prior work with the platform tools is expected, although some knowledge of Kubernetes helps.