Module 00: Elasticsearch Fundamentals

Why this matters

In track 04 you learned to model data in a relational database and ask it precise questions with SQL. Relational databases are superb at exactly what they're built for: structured data, transactions, joins, strong consistency. But there's a whole class of question they answer badly, and the moment your product grows a search box you run straight into it.

Try this in Postgres: "find every product whose description mentions running shoes, rank the best matches first, tolerate the user typing runing shoez, and highlight the matching words." The naive tool is WHERE description LIKE '%running shoes%'. That query (a) can't use a normal B-tree index so it scans every row, (b) only matches the exact substring in that exact order, (c) has no concept of "best match" — every row either matches or doesn't, there's no ranking, and (d) has no idea runing was meant to be running. You can bolt on Postgres full-text search (tsvector/tsquery) and get some of this, and for modest needs you should. But once you want real relevance ranking, typo tolerance, faceted filtering, autocomplete, and log/analytics aggregations over hundreds of millions of documents, you reach for a purpose-built search engine.

That engine, for most of the industry, is Elasticsearch. This module is the conceptual foundation: why it exists, and the two or three ideas — the inverted index, relevance scoring, shards and segments — that make everything in the rest of this track make sense. Skipping the concepts here is the single most common reason people find Elasticsearch behaving "magically" later. It isn't magic; it's a small number of mechanical ideas applied consistently.

Concepts

What Elasticsearch is (and what it sits on top of)

Elasticsearch is a distributed, document-oriented search and analytics engine. Three words worth unpacking:

• Document-oriented: you store JSON documents, not rows in fixed tables. A document is roughly analogous to a row, an index is roughly analogous to a table, and a field is roughly a column — but the analogy leaks (there are no joins in the SQL sense, and the same field can be stored several ways at once; more on that in module 01).
• Search and analytics: it's optimized for finding documents by content and aggregating over them, not for transactional bookkeeping. It is not a system of record. The canonical architecture is: your relational database (track 04) remains the source of truth, and you index a copy of the searchable data into Elasticsearch. If Elasticsearch loses data, you rebuild it from the database. Never make Elasticsearch the only home of data you can't afford to lose.
• Distributed: it's designed from the ground up to spread data across many machines (nodes) and keep serving if some fail. That's why the vocabulary is full of shards and replicas.

Under the hood, Elasticsearch is a distributed wrapper around Apache Lucene, a mature Java search library. Lucene does the actual work of building inverted indexes, scoring documents, and reading/writing the on-disk data structures for a single index on a single machine. Lucene by itself is just a library you'd embed in a program — it has no network API, no clustering, no JSON. Elasticsearch wraps Lucene with: a REST/JSON API, a query language (Query DSL), and — crucially — the distribution layer that shards data across nodes and replicates it for fault tolerance. When you read that "a shard is a Lucene index," that's literally true: each shard is one self-contained Lucene index. Keep this relationship in your head — a surprising amount of Elasticsearch behavior is really Lucene behavior showing through.

The inverted index: the one idea that makes search fast

Here is the mechanism that separates a search engine from a LIKE '%...%' scan. Suppose you have three documents:

doc 1: "the quick brown fox"
doc 2: "the lazy brown dog"
doc 3: "quick foxes jump"

A relational LIKE scan stores these as strings and, to answer "which docs contain brown?", reads every document and checks. That's O(number of docs).

An inverted index flips the data structure inside out. At index time, Elasticsearch breaks each document's text into terms (tokens) and builds a map from term → list of documents containing it:

term      → postings list (which docs, and where)
--------------------------------------------------
brown     → [doc 1, doc 2]
dog       → [doc 2]
fox       → [doc 1]
foxes     → [doc 3]
jump      → [doc 3]
lazy      → [doc 2]
quick     → [doc 1, doc 3]
the       → [doc 1, doc 2]

Now "which docs contain brown?" is a single lookup of the term brown — you jump straight to [doc 1, doc 2] without touching the other documents. This is why search over 100 million documents can return in milliseconds: the query cost scales with the number of matching terms, not the number of documents. It's the same reason the index at the back of a textbook lets you find every mention of "photosynthesis" without reading the whole book — an inverted index is exactly a book index, built automatically.

The terms in the index are the unit of matching. A query only matches if its terms match the terms in the index — and both go through a text-processing step called analysis (module 02) that decides what a "term" is. This is the root of the single most common Elasticsearch confusion: you search for Running and get nothing, because the indexed term was lowercased to running and your query wasn't analyzed the same way. Hold that thought; it's the payoff of module 02.

Relevance: TF-IDF and BM25 at a conceptual level

Because a search query can match thousands of documents, the engine must rank them — put the most relevant first. Relevance is a computed number, the _score, and understanding roughly how it's computed demystifies why one result beats another.

The classic intuition is TF-IDF, which combines two signals:

• Term Frequency (TF) — how often the search term appears in this document. A document that says "shoes" ten times is probably more about shoes than one that says it once. More is better (with diminishing returns).
• Inverse Document Frequency (IDF) — how rare the term is across the whole index. A match on a rare word like "photosynthesis" tells you far more than a match on a ubiquitous word like "the." Rarer terms carry more weight; common terms carry almost none.

Multiply them and you get the core idea: a document scores high when it contains the query's terms often (TF) and those terms are rare across the corpus (IDF). A match on a distinctive term in a document that emphasizes it ranks above a document that mentions a common term once.

Modern Elasticsearch doesn't use raw TF-IDF; it uses BM25, a refined version of the same two signals. BM25 adds two important corrections: saturation (the 20th occurrence of a term barely adds more than the 10th — TF has diminishing returns instead of growing without bound) and length normalization (a term appearing 5 times in a 20-word document is more significant than 5 times in a 5,000-word document). You'll meet BM25 properly in module 03; for now the takeaway is just: _score is TF-style "how much this doc is about the term" times IDF-style "how rare/informative the term is," tuned so long documents and repeated words don't dominate unfairly.

  _score(doc, "photosynthesis") is nudged by three factors:

    TF   how often the term appears in THIS doc ──► more  = higher (saturates)
     ×
    IDF  how rare the term is across ALL docs   ──► rarer = higher
     ×
    len  short field for the same match         ──► shorter = higher

Shards and replicas: how one index spans many machines

An Elasticsearch index is not one big file. It's split into shards, and each shard is a full, independent Lucene index holding a subset of the documents. Two reasons this matters:

• Scale: a single machine can only hold and search so much data. Splitting an index into, say, 5 shards lets Elasticsearch put those shards on 5 different nodes, so both storage and query work spread across the cluster. A search hits every shard in parallel and the results are merged.
• The number of primary shards is (effectively) fixed at index creation. You choose it when you create the index and you can't cheaply change it later — you'd have to reindex into a new index. Choosing badly is a real, common mistake, in both directions. Too few shards and you can't spread a huge index across nodes. Too many shards — the far more common beginner error — and each shard is tiny, but every shard carries fixed overhead (its own Lucene structures, memory, and per-query coordination cost), so a query fans out to dozens of near-empty shards and the coordination overhead dominates. You'll see this "over-sharding" performance trap again in module 05. For small datasets, one primary shard is often the right answer, and that's the default in modern Elasticsearch.

  index "books"  (1 primary + 1 replica)      single-node cluster
  ┌─────────────┐                             ┌──────── node 1 ────────┐
  │ primary  P0 │ ─────────────────────────►  │  P0  STARTED           │
  │ replica  R0 │ ─ must live on ANOTHER node │  R0  no other node ────┼─► UNASSIGNED
  └─────────────┘                             └────────────────────────┘  → health YELLOW

Replicas are copies of a shard. A replica shard is a redundant copy of a primary shard, kept on a different node. Replicas do two jobs: fault tolerance (if the node holding a primary dies, a replica is promoted to primary and no data is lost) and read throughput (searches can be served by either the primary or its replicas, so more replicas means more concurrent search capacity). Replicas can be changed on a live index, unlike primary shard count. Note: on a single-node cluster (like the Docker setup you'll run in this track), replicas can't be allocated — there's no other node to put them on — so the cluster health goes yellow. Yellow on a single node is normal and expected, not a problem to fix; you'll see it constantly and module 06 explains it fully.

Segments: why documents aren't instantly searchable, and why deletes are lazy

Inside a single shard (one Lucene index), the data is stored in segments — smaller immutable sub-indexes. This detail explains two behaviors that otherwise look like bugs:

• Near-real-time, not real-time. When you index a document, it isn't searchable the instant the API call returns. New documents accumulate in an in-memory buffer and become searchable only when a refresh flushes them into a new searchable segment — by default about once per second. So a document you just indexed may not appear in a search for up to a second. This trips people up constantly in tests ("I indexed it, why can't I find it?"). You can force a refresh (?refresh=true) in tests, but you should not do that in production hot paths — it's expensive.
• Segments are immutable; deletes and updates are lazy. Because a segment, once written, is never modified, you can't edit a document in place. An "update" is really index a new version + mark the old one deleted, and a delete just marks the document as deleted (a tombstone) without removing it from the segment. The space is reclaimed later during merging, a background process that combines small segments into larger ones and physically drops the deleted docs. This is why a freshly "cleared" index can still show disk usage, and why heavy update/delete workloads generate merge activity.

When to reach for Elasticsearch — and when not to

Elasticsearch earns its keep for:

• Full-text search with relevance ranking (product catalogs, documentation, content sites) — the LIKE problem done properly.
• Log and event analytics — ingesting huge volumes of logs/metrics and aggregating them (this is the "ELK/Elastic stack" you may have heard of; it pairs with Kibana, module 06).
• Autocomplete / type-ahead — suggesting completions as a user types.
• Faceted search / aggregations — "show counts by brand, price bucket, rating" alongside results, computed fast over the whole result set.

Reach for something else when:

• You need transactions and strong consistency (money, inventory decrements) — that's your relational database's job. Elasticsearch has no multi-document ACID transactions.
• The data is your only copy and must never be lost — Elasticsearch is a secondary index over a source of truth, not the source of truth.
• Your query needs are simple exact-key lookups — a database or key-value store is simpler and cheaper.

The healthy mental model for this whole track: your relational database from track 04 stays the system of record, and Elasticsearch is a fast, denormalized, searchable projection of the parts of that data users need to search.

Command reference

You won't run many real queries in this conceptual module — the goal is to stand up a cluster and confirm you can talk to it. But here's the reference you'll build on for the whole track.

Bring up a single-node Elasticsearch and Kibana with Docker Compose. Save this as docker-compose.yml — you'll reuse it for the whole track:

# docker-compose.yml — single-node Elasticsearch + Kibana for local learning
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.13.4
    container_name: es
    environment:
      - discovery.type=single-node        # don't try to form a multi-node cluster
      - xpack.security.enabled=false      # no auth/TLS — LOCAL LEARNING ONLY
      - ES_JAVA_OPTS=-Xms1g -Xmx1g        # cap heap so it fits a laptop
    ports:
      - "9200:9200"
    healthcheck:
      test: ["CMD-SHELL", "curl -s http://localhost:9200 >/dev/null || exit 1"]
      interval: 10s
      timeout: 5s
      retries: 12

  kibana:
    image: docker.elastic.co/kibana/kibana:8.13.4
    container_name: kibana
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    ports:
      - "5601:5601"
    depends_on:
      elasticsearch:
        condition: service_healthy

Security note: xpack.security.enabled=false disables authentication and TLS. That's fine for a throwaway local cluster on your laptop and keeps the exercises focused on search, but never run a production or internet-reachable cluster this way — an open Elasticsearch is one of the classic sources of public data leaks.

The Python client (elasticsearch-py) — install and connect:

pip install "elasticsearch>=8,<9"

from elasticsearch import Elasticsearch

# Matches the docker-compose above (no auth, plain HTTP, localhost)
es = Elasticsearch("http://localhost:9200")

print(es.info())                       # cluster name + version
print(es.cluster.health())             # dict with 'status', 'number_of_nodes', ...

Hands-on exercises

You need Docker running (from track 02 / learn/02-docker). All exercises use the docker-compose.yml above.

1. Bring up the cluster

docker compose up -d
docker compose ps

Expected: two containers, es and kibana. Elasticsearch takes ~30-60s to become healthy on first start; Kibana takes a minute or two more (it waits for Elasticsearch). Watch logs with docker compose logs -f elasticsearch until you see a line about the node having started.

2. Confirm you can talk to it

curl localhost:9200

Expected: a JSON blob with "cluster_name", "version": {"number": "8.13.4", ...}, and the tagline "You Know, for Search". If you get connection refused, the container isn't healthy yet — wait and retry.

3. Read the cluster health and interpret the color

curl "localhost:9200/_cluster/health?pretty"

Expected: "status" is almost certainly "yellow", "number_of_nodes": 1. Write down why it's yellow before reading on. (Answer: there's one node, so replica shards have nowhere to go — a replica can't sit on the same node as its primary. That's expected on a single-node cluster and is not an error.)

4. Create an index and watch it in _cat

curl -XPUT "localhost:9200/books"
curl "localhost:9200/_cat/indices?v"

Expected: a books index listed with health yellow, pri 1 (one primary shard, the modern default), rep 1 (one replica — the one that can't be allocated, hence yellow). Note docs.count 0.

5. Index your first documents

curl -XPOST "localhost:9200/books/_doc" -H 'Content-Type: application/json' -d '{
  "title": "The Quick Brown Fox",
  "year": 2019
}'
curl -XPOST "localhost:9200/books/_doc" -H 'Content-Type: application/json' -d '{
  "title": "Lazy Dog Stories",
  "year": 2021
}'

Expected: each returns JSON with "result": "created", an auto-generated "_id", and "_version": 1.

6. Observe near-real-time: the one-second gap

Index a document and immediately count — you're racing the refresh:

curl -XPOST "localhost:9200/books/_doc" -H 'Content-Type: application/json' -d '{"title":"Instant Book","year":2024}'
curl "localhost:9200/books/_count"

Run the _count a few times quickly. You may see the count not include the new doc for up to ~1 second, then jump. That gap is the refresh interval — new docs sit in a buffer until the next refresh turns them into a searchable segment. This is "near-real-time" in action.

7. Force a refresh and confirm

curl -XPOST "localhost:9200/books/_refresh"
curl "localhost:9200/books/_count"

Expected: the count now definitely includes all indexed docs. In real code you rarely force refreshes (it's costly); in tests you often do so results are deterministic.

8. Do the same from Python

from elasticsearch import Elasticsearch
es = Elasticsearch("http://localhost:9200")

es.index(index="books", document={"title": "Python and the Fox", "year": 2020})
es.indices.refresh(index="books")
print(es.count(index="books"))     # {'count': 4, ...}
print(es.cluster.health()["status"])   # 'yellow'

Expected: the count reflects everything indexed so far, and health prints yellow.

9. Diagnose and fix: "I indexed it but search finds nothing"

A teammate runs this and is convinced Elasticsearch is broken:

curl -XPOST "localhost:9200/nrt-demo/_doc" -H 'Content-Type: application/json' -d '{"msg":"hello"}'
curl "localhost:9200/nrt-demo/_search?q=hello"   # returns 0 hits!

Reproduce it (run both lines back-to-back, fast). Explain what's happening and give two fixes.

10. Tear down (or leave it running)

docker compose down            # stops and removes containers (data in the
                               # anonymous volume is discarded)

If you want to keep going straight into module 01, leave it up instead. You'll use this same cluster for the entire track.

Independent challenge

No commands given — reason it through using this module's concepts.

Task: Without looking anything up, predict the following and then verify each against your running cluster with _cat and _cluster/health:

1. You create an index logs-2026 with 3 primary shards and 1 replica on your single-node Docker cluster. What color will the cluster health be, and exactly how many shards will be unassigned, and why?
2. You then start a second Elasticsearch node (imagine it — or reason about it). What happens to the health color and the unassigned shards, and why?
3. You index 50 documents into logs-2026. Roughly how are they distributed across the 3 primary shards, and does a search have to touch all 3?

Connect this back to the "Shards and replicas" concept above — in particular, why the replica shards can't be assigned on a single node, and what a replica is actually for.

Common mistakes & troubleshooting

• Treating Elasticsearch as a system of record. It's a secondary, rebuildable index over a source of truth (your database). No ACID transactions; design so you can always reindex from the database.
• Panicking about yellow health on a single node. Yellow just means replicas are unassigned because there's only one node. Expected. Red is the one to worry about (a primary shard is missing → actual data unavailable).
• "I indexed it but can't find it." Near-real-time refresh gap (~1s). Wait or force a refresh; don't conclude the write failed.
• Over-sharding small indices. Every shard has fixed overhead; many tiny shards is slower, not faster. For small data, one primary shard is fine. Primary shard count is fixed at creation — choose deliberately.
• Expecting LIKE-style substring behavior. Search matches terms produced by analysis, not raw substrings. Running vs running vs runners behave according to the analyzer, not string equality — the whole point of modules 01-02.
• Running with security disabled anywhere but a throwaway local box. Fine for these exercises; a serious mistake if exposed to a network.

Checkpoint quiz

Write down your answer to each question before expanding it — checking without attempting first is the single easiest way to fool yourself into thinking you've learned this.

1. In one or two sentences, what problem does an inverted index solve, and why is it dramatically faster than WHERE description LIKE '%term%'?
2. What is the relationship between Elasticsearch and Apache Lucene? Where does "a shard is a Lucene index" fit in?
3. Name the two signals TF-IDF combines and, in plain English, what kind of document scores highest.
4. What are the two jobs a replica shard does, and why does a single-node cluster show yellow health?
5. Why isn't a document searchable the instant you index it, and what is the name of the process that makes it searchable?
6. Give one scenario where you should use Elasticsearch and one where you should not, with the reason for each.

Further reading & sources

• Elasticsearch: What is Elasticsearch? - the official orientation on what the engine is and the problems it targets.
• Elasticsearch: The Definitive Guide — Inverted Index - the canonical worked explanation of the data structure this module is built around.
• Data in: documents and indices - how documents, indices, shards, and near-real-time refresh relate.
• Practical BM25 (Part 2): The BM25 algorithm and its variables - Elastic's plain-language walkthrough of TF, IDF, saturation, and length normalization.
• Apache Lucene - the underlying search library each Elasticsearch shard is an instance of.
• Scalability and resilience: clusters, nodes, and shards - how sharding and replication give scale and fault tolerance.

Next

01-indexes-and-mappings — now that you know why Elasticsearch exists and how it stores data conceptually, you'll learn to define the structure of an index: field mappings, the pivotal text vs keyword distinction, and the dynamic-mapping pitfalls that quietly wreck relevance and aggregations if you let Elasticsearch guess.