diff --git a/lessons/en/storage/cachex.md b/lessons/en/storage/cachex.md new file mode 100644 index 0000000000..ef67bd54cb --- /dev/null +++ b/lessons/en/storage/cachex.md @@ -0,0 +1,380 @@ +%{ + version: "1.0.0", + title: "Cachex", + excerpt: """ + Cachex is a powerful caching library for Elixir with support for transactions, fallbacks, expirations, and distributed operations. + In this lesson, we'll learn how to use Cachex to improve application performance through intelligent caching strategies. + """ +} +--- + +## Overview + +Cachex is a feature-rich in-memory caching library for Elixir built on top of ETS. It provides automatic expiration with configurable TTL policies, distributed caching across multiple nodes, and transactional operations for atomic cache updates. Fallback functions handle lazy loading of missing data, batch operations improve performance, and hooks give us metrics, monitoring, and custom behavior. + +## Installation and Setup + +To add Cachex to our project let's add it to our `mix.exs` dependencies: + +```elixir +defp deps do + [{:cachex, "~> 4.1"}] +end +``` + +Then we need to install our dependencies: + +```shell +mix deps.get +``` + +## Basic Usage + +### Starting a Cache + +For quick testing in IEx, you can start a cache manually: + +```elixir +iex> Cachex.start_link(:my_cache) +{:ok, #PID<0.123.0>} +``` + +For production applications, it's best to add Cachex to our supervision tree in `application.ex`: + +```elixir +children = [ + {Cachex, [:my_cache, []]} +] +``` + +### Core Operations + +Cachex provides intuitive functions for basic cache operations: + +```elixir +iex> {:ok, _pid} = Cachex.start_link(:my_cache) + +# Store a value +iex> {:ok, true} = Cachex.put(:my_cache, "user:123", %{name: "Alice", age: 30}) + +# Check if a key exists +iex> {:ok, true} = Cachex.exists?(:my_cache, "user:123") + +# Retrieve a value +iex> {:ok, %{name: "Alice", age: 30}} = Cachex.get(:my_cache, "user:123") + +# Delete a key +iex> {:ok, true} = Cachex.del(:my_cache, "user:123") +``` + +### Unsafe Operations + +Cachex provides "unsafe" versions of functions (suffixed with `!`) that unpack tuples and raise on errors: + +```elixir +# Safe version returns tuples +iex> {:ok, nil} = Cachex.get(:my_cache, "missing_key") + +# Unsafe version returns values directly +iex> nil = Cachex.get!(:my_cache, "missing_key") + +# Errors raise exceptions with unsafe versions +iex> Cachex.get!(:nonexistent_cache, "key") +** (Cachex.Error) Specified cache not running +``` + +These are convenient for testing and chaining operations but be careful using unsafe functions in production where explicit error handling is preferred. + +## Advanced Operations + +### Batch Operations + +For better performance when dealing with multiple keys, use batch operations: + +```elixir +iex> {:ok, _pid} = Cachex.start_link(:my_cache) + +iex> {:ok, true} = Cachex.put_many(:my_cache, [ +...> {"user:1", %{name: "Alice"}}, +...> {"user:2", %{name: "Bob"}}, +...> {"user:3", %{name: "Charlie"}} +...> ]) +``` + +### Atomic Operations + +Cachex supports atomic updates for safe concurrent modifications: + +```elixir +iex> Cachex.put(:my_cache, "counter", 0) +iex> {:ok, 1} = Cachex.incr(:my_cache, "counter") +iex> {:ok, 2} = Cachex.incr(:my_cache, "counter") + +iex> {:ok, 1} = Cachex.decr(:my_cache, "counter") + +iex> {:commit, 10} = Cachex.get_and_update(:my_cache, "counter", fn value -> +...> value * 10 +...> end) +``` + +## Expiration and TTL + +### Default Expiration + +Cachex provides the ability for us to set a default expiration for all cache entries: + +```elixir +import Cachex.Spec + +Cachex.start_link(:my_cache, [ + expiration: expiration(default: :timer.minutes(5)) +]) +``` + +### Per-Key Expiration + +Alternatively, we can set an expiration when storing values: + +```elixir +iex> Cachex.put(:my_cache, "session:abc", user_data, expire: :timer.seconds(60)) + +iex> Cachex.put(:my_cache, "key", "value") +iex> Cachex.expire(:my_cache, "key", :timer.seconds(30)) +``` + +## Lazy Loading with Fetch + +The `fetch/4` function provides elegant lazy loading when keys are missing. When using `fetch/4` Cachex will execute our function, and cache the result, whenever there is no value present at the key: + +```elixir +{:commit, 6} = Cachex.fetch(:my_cache, "tarzan", &String.length/1) +``` + +It's possible to set an expiration when using `fetch/4`: + +```elixir +{:commit, data} = Cachex.fetch(:my_cache, "api:users", fn -> + users = fetch_users_from_api() + {:commit, users, expire: :timer.minutes(10)} +end) +``` + +The `fetch/4` function is particularly powerful for preventing cache stampedes in concurrent environments. Unlike manual cache-miss handling, `fetch/4` ensures that concurrent requests for the same missing key will queue behind a single computation: + +```elixir +# Multiple concurrent requests for the same missing key +for _ <- 1..10 do + spawn(fn -> + Cachex.fetch(:cache, "expensive_key", fn -> + # This expensive operation runs only once + expensive_database_call() + end) + end) +end +``` + +### Optimized Batch Execution + +We can use `Cachex.execute!/2` to efficiently perform several operations in a single function call; everything runs inside the same process without round trips between our code and the cache, minimizing messaging overhead and locking. + +```elixir +{r1, r2, r3} = Cachex.execute!(:my_cache, fn cache -> + r1 = Cachex.get!(cache, "key1") + r2 = Cachex.get!(cache, "key2") + r3 = Cachex.get!(cache, "key3") + {r1, r2, r3} +end) +``` + +## Transactions + +While `Cachex.execute!/3` is great for batch operations, `Cachex.transaction!/3` ensures that all operations within the transaction are atomic and that the specified keys are properly locked during execution. + +```elixir +result = Cachex.transaction!(:my_cache, ["user:1", "user:2"], fn cache -> + user1 = Cachex.get!(cache, "user:1") + user2 = Cachex.get!(cache, "user:2") + + updated_user1 = update_user(user1) + + Cachex.put!(cache, "user:1", updated_user1) + + updated_user1 +end) +``` + +**Important**: You must specify all keys that will be accessed in the transaction. This ensures proper locking and prevents deadlocks. + +## Statistics and Monitoring + +Using hooks, we can enable statistics collection in Cachex. Let's look at an example scenario to see what statistics are available: + +```elixir +import Cachex.Spec + +Cachex.start_link(:my_cache, [ + hooks: [ + hook(module: Cachex.Stats) + ] +]) + +Cachex.put!(:my_cache, "key1", "value1") +Cachex.put!(:my_cache, "key2", "value2") +Cachex.get!(:my_cache, "key1") # hit +Cachex.get!(:my_cache, "key3") # miss + +stats = Cachex.stats!(:my_cache) +IO.inspect(stats) + +# %{ +# meta: %{creation_date: 1726777631670}, +# hits: 1, +# misses: 1, +# hit_rate: 50.0, +# miss_rate: 50.0, +# calls: %{get: 2, put: 2}, +# operations: 4, +# writes: 2 +# } +``` + +We see in our output there's some valuable information available to us: the number of hits and misses, the percentage of each, how many of each call has been performed, the total number of operations, and the number of writes. + +## Cache Limiting and Pruning + +### Automatic Limiting with Hooks + +Another ability of hooks is tracking access times and preventing memory issues by limiting cache size: + +```elixir +import Cachex.Spec + +Cachex.start_link(:my_cache, [ + hooks: [ + hook(module: Cachex.Limit.Accessed), # Track access times + hook(module: Cachex.Limit.Scheduled, args: { + 500, # Maximum 500 entries + [], # Options for Cachex.prune/3 + [] # Options for Cachex.Limit.Scheduled + }) + ] +]) +``` + +## Persistence + +### Saving and Restoring Cache Data + +Cachex can persist cache data to disk and restore from disk, automatically merging with existing data: + +```elixir +{:ok, true} = Cachex.save(:my_cache, "/tmp/my_cache.dat") + +{:ok, _count} = Cachex.restore(:my_cache, "/tmp/my_cache.dat") +``` + +**Note**: Expired entries in the saved file will not be restored. + +## Distributed Caching + +Cachex supports distributed caching across multiple nodes using routers: + +```elixir +import Cachex.Spec + +Cachex.start_link(:distributed_cache, [ + router: router(module: Cachex.Router.Ring, options: [ + monitor: true + ]) +]) +``` + +Some things to keep in mind when we're working in distributed mode: +- Keys are automatically routed to appropriate nodes +- Multi-key operations require keys to be on the same node +- Some operations like `stream/3` are not available in distributed mode + +## Custom Hooks + +With custom hooks we can extend Cachex's functionality: + +```elixir +defmodule MyApp.CacheLogger do + use Cachex.Hook + require Logger + + def init(_), do: {:ok, nil} + + def handle_notify({action, _args}, result, state) do + Logger.info("Cache #{action}: #{inspect(result)}") + {:ok, state} + end +end + +Cachex.start_link(:my_cache, [ + hooks: [ + hook(module: MyApp.CacheLogger) + ] +]) +``` + +## Real-World Example + +Let's look at a practical example of using Cachex in a web application. In this example we'll limit the size of our cache to 1000 entries and set an hour TTL on entries. We'll use the `fetch/4` function we looked at previously to retrieve our cached user or look the user up in the event there is a cache miss. When we update our user's record, we'll update our cache record: + +```elixir +defmodule MyApp.UserCache do + @cache_name :user_cache + + def start_link do + import Cachex.Spec + + Cachex.start_link(@cache_name, [ + expiration: expiration(default: :timer.hours(1)), + hooks: [ + hook(module: Cachex.Stats), + hook(module: Cachex.Limit.Scheduled, args: {1000, [], []}) + ] + ]) + end + + def get_user(user_id) do + Cachex.fetch(@cache_name, "user:#{user_id}", fn -> + case MyApp.Users.get_user(user_id) do + {:ok, user} -> {:commit, user} + {:error, :not_found} -> {:ignore, nil} + end + end) + end + + def update_user(user_id, attrs) do + with {:ok, user} <- MyApp.Users.update_user(user_id, attrs) do + Cachex.put(@cache_name, "user:#{user_id}", user) + {:ok, user} + end + end + + def invalidate_user(user_id) do + Cachex.del(@cache_name, "user:#{user_id}") + end + + def stats do + Cachex.stats(@cache_name) + end +end +``` + +## Best Practices + +Caching can be a powerful tool in the toolbox but there's a few things to consider when using it to ensure we achieve the best results: + ++ Use TTLs to balance performance with data freshness requirements, it's a good idea to pair TTLs with proper fallback strategies using `fetch/4`. ++ If we need to work with multiple keys at the same time use batch operations in Cachex. ++ Transactions are essential for operations requiring consistency across multiple keys. ++ Use Cachex's distributed caching in a multi-node environment for scalability and fault tolerance. + +## Conclusion + +Cachex provides a comprehensive caching solution for Elixir applications, offering everything from simple key-value storage to advanced distributed caching with transactions and monitoring. Its rich feature set and extensible architecture make it suitable for applications of any scale, whether we're building simple web apps or large distributed systems. + +The library's emphasis on performance, reliability, and developer experience makes it an excellent choice for improving application performance through intelligent caching strategies. Whether we need basic memoization or complex distributed cache coordination, Cachex provides the tools we need. \ No newline at end of file diff --git a/lessons/en/storage/ets.md b/lessons/en/storage/ets.md index 8ef821b568..0ddd3a09a4 100644 --- a/lessons/en/storage/ets.md +++ b/lessons/en/storage/ets.md @@ -1,87 +1,161 @@ %{ - version: "1.1.1", + version: "2.0.0", title: "Erlang Term Storage (ETS)", excerpt: """ - Erlang Term Storage, commonly referred to as ETS, is a powerful storage engine built into OTP and available to use in Elixir. - In this lesson we'll look at how to interface with ETS and how it can be employed in our applications. + Erlang Term Storage, commonly referred to as ETS, is a powerful in-memory storage engine built into the Erlang VM and available to use in Elixir. + In this lesson we'll explore how to interface with ETS and how it can be employed in our applications for high-performance data storage and retrieval. """ } --- ## Overview -ETS is a robust in-memory store for Elixir and Erlang objects that comes included. -ETS is capable of storing large amounts of data and offers constant time data access. +ETS is a robust in-memory storage system for Elixir and Erlang terms that comes included with the runtime capable of storing large amounts of data and offers constant time data access for most operations. It's particularly useful for caching, lookups, and scenarios where we need fast access to structured data. Tables in ETS are created and owned by individual processes. -When an owner process terminates, its tables are destroyed. -You can have as many ETS table as you want, the only limit is the server memory. A limit can be specified using the `ERL_MAX_ETS_TABLES` environment variable. +When an owner process terminates, its tables are automatically destroyed. +By default, we can create as many ETS tables as memory allows, though we can set limits using the `ERL_MAX_ETS_TABLES` environment variable. + +ETS offers constant time lookups for most operations, concurrent access from multiple processes, and flexible storage with different table types to suit various use cases. Data is stored directly in the VM without serialization overhead, keeping both memory usage and access times low. ## Creating Tables -Tables are created with `new/2`, which accepts a table name, and a set of options, and returns a table identifier that we can use in subsequent operations. +Tables are created with `:ets.new/2`, which accepts a table name and a set of options, returning a table identifier for subsequent operations. -For our example we'll create a table to store and look up users by their nickname: +For our example, we'll create a table to store and look up users by their nickname: ```elixir iex> table = :ets.new(:user_lookup, [:set, :protected]) -8212 +#Reference<0.1234567890.1234567890.123456> ``` -Much like GenServers, there is a way to access ETS tables by name rather than identifier. -To do this we need to include the `:named_table` option. -Then we can access our table directly by name: +Much like GenServers, there's a way to access ETS tables by name rather than identifier. +To do this we need to include the `:named_table` option: ```elixir iex> :ets.new(:user_lookup, [:set, :protected, :named_table]) :user_lookup ``` +Now we can access our table directly by name instead of keeping track of the reference. + ### Table Types -There are four types of tables available in ETS: +ETS provides four different table types to suit various use cases: + ++ The default `set` type stores one value per key with unique keys, making it ideal for simple key-value storage. ++ An `ordered_set` works similarly to `set` but maintains order by Erlang/Elixir term ordering, where key comparison uses Erlang's term ordering and `1` and `1.0` are considered equal. ++ The `bag` type allows multiple objects per key but only permits one instance of each unique object per key. ++ Finally, `duplicate_bag` supports multiple objects per key with duplicates allowed, useful for scenarios like storing multiple events for the same timestamp. + +Let's see the differences in action: -+ `set` — This is the default table type. -One value per key. -Keys are unique. -+ `ordered_set` — Similar to `set` but ordered by Erlang/Elixir term. -It is important to note that key comparison is different within `ordered_set`. -Keys need not match so long as they compare equally. -1 and 1.0 are considered equal. -+ `bag` — Many objects per key but only one instance of each object per key. -+ `duplicate_bag` — Many objects per key, with duplicates allowed. +#### Set + +```elixir +iex> :ets.new(:set_example, [:set, :named_table]) +iex> :ets.insert(:set_example, {:key, "value1"}) +iex> :ets.insert(:set_example, {:key, "value2"}) # Overwrites previous +iex> :ets.lookup(:set_example, :key) +[{:key, "value2"}] +``` + +#### Bag + +```elixir +iex> :ets.new(:bag_example, [:bag, :named_table]) +iex> :ets.insert(:bag_example, {:key, "value1"}) +iex> :ets.insert(:bag_example, {:key, "value2"}) # Adds to existing +iex> :ets.insert(:bag_example, {:key, "value2"}) # Ignored +iex> :ets.lookup(:bag_example, :key) +[{:key, "value1"}, {:key, "value2"}] +``` + +#### Duplicate Bag + +```elixir +iex> :ets.new(:duplicate_bag_example, [:duplicate_bag, :named_table]) +iex> :ets.insert(:duplicate_bag_example, {:key, "value1"}) +iex> :ets.insert(:duplicate_bag_example, {:key, "value2"}) +iex> :ets.insert(:duplicate_bag_example, {:key, "value2"}) +iex> :ets.lookup(:duplicate_bag_example, :key) +[{:key, "value1"}, {:key, "value2"}, {:key, "value2"}] # Two instances of `{:key, "value2"}` exist +``` ### Access Controls -Access control in ETS is similar to access control within modules: +Access control in ETS determines which processes can read from and write to our tables, there are three modes: `public`, `protected`, and `private`. + ++ The `public` mode makes read and write operations available to all processes, which is useful for shared data structures but requires careful coordination to avoid race conditions. ++ The `protected` mode, which is the default, allows read access to all processes while restricting write access to only the owner process, providing a good balance of accessibility and safety. ++ The `private` mode limits both read and write access to the owner process only, offering maximum isolation at the cost of reduced accessibility. + +```elixir +iex> public_table = :ets.new(:public_example, [:set, :public, :named_table]) +iex> protected_table = :ets.new(:protected_example, [:set, :protected, :named_table]) +iex> private_table = :ets.new(:private_example, [:set, :private, :named_table]) +``` + +## Concurrency and Race Conditions + +When multiple processes can write to a table (via `:public` access or by sending messages to the owning process), race conditions are possible. +For example, two processes each reading a counter value of `0`, incrementing it, and writing `1` back would result in a lost increment. + +For counters specifically, `:ets.update_counter/3` provides atomic update-and-read operations: + +```elixir +iex> :ets.new(:counters, [:set, :public, :named_table]) +iex> :ets.insert(:counters, {:page_views, 0}) +iex> :ets.update_counter(:counters, :page_views, 1) +1 +iex> :ets.update_counter(:counters, :page_views, 5) +6 +``` + +For other atomic operations, we might need the owner process to handle updates through message passing to ensure consistency. -+ `public` — Read/Write available to all processes. -+ `protected` — Read available to all processes. -Only writable by owner process. -This is the default. -+ `private` — Read/Write limited to owner process. +## Performance Features -## Race Conditions +### Write Concurrency -If more than one process can write to a table - whether via `:public` access or by messages to the owning process - race conditions are possible. -For example, two processes each read a counter value of `0`, increment it, and write `1`; the end result reflects only a single increment. +ETS tables can be optimized for concurrent writes using the `write_concurrency` option: -For counters specifically, [:ets.update_counter/3](http://erlang.org/doc/man/ets.html#update_counter-3) provides for atomic update-and-read. -For other cases, it may be necessary for the owner process to perform custom atomic operations in response to messages, such as "add this value to the list at key `:results`". +```elixir +iex> :ets.new(:concurrent_table, [:set, :public, {:write_concurrency, true}]) +``` -## Inserting data +Starting with OTP 25, we can use `{:write_concurrency, :auto}` to let the runtime automatically optimize based on usage patterns: + +```elixir +iex> :ets.new(:adaptive_table, [:set, :public, {:write_concurrency, :auto}]) +``` -ETS has no schema. -The only limitation is that data must be stored as a tuple whose first element is the key. -To add new data we can use `insert/2`: +### Read Concurrency + +For tables with many concurrent readers, enable read concurrency: + +```elixir +iex> :ets.new(:read_heavy_table, [:set, :public, {:read_concurrency, true}]) +``` + +These options aren't mutually exclusive — we can enable both for tables with heavy concurrent reads and writes: + +```elixir +iex> :ets.new(:busy_table, [:set, :public, {:write_concurrency, :auto}, {:read_concurrency, true}]) +``` + +## Inserting Data + +ETS objects have no predefined schema - the only requirement is that data must be stored as tuples where the first element is the key. +To add a new object we use `:ets.insert/2`: ```elixir iex> :ets.insert(:user_lookup, {"doomspork", "Sean", ["Elixir", "Ruby", "Java"]}) true ``` -When we use `insert/2` with a `set` or `ordered_set` existing data will be replaced. -To avoid this there is `insert_new/2` which returns `false` for existing keys: +When using `:ets.insert/2` with a `set` or `ordered_set`, existing object with the same key will be replaced. +To avoid overwriting, use `:ets.insert_new/2` which returns `false` for existing keys: ```elixir iex> :ets.insert_new(:user_lookup, {"doomspork", "Sean", ["Elixir", "Ruby", "Java"]}) @@ -90,40 +164,55 @@ iex> :ets.insert_new(:user_lookup, {"3100", "", ["Elixir", "Ruby", "JavaScript"] true ``` -## Data Retrieval +We can also insert multiple objects at once: + +```elixir +iex> users = [ +...> {"alice", "Alice", ["Python", "Go"]}, +...> {"bob", "Bob", ["JavaScript", "TypeScript"]}, +...> {"charlie", "Charlie", ["Rust", "C++"]} +...> ] +iex> :ets.insert(:user_lookup, users) +true +``` -ETS offers us a few convenient and flexible ways to retrieve our stored data. -We'll look at how to retrieve data by key and through different forms of pattern matching. +## Data Retrieval -The most efficient, and ideal, retrieval method is key lookup. -While useful, matching iterates through the table and should be used sparingly especially for very large data sets. +ETS offers several convenient and flexible ways to retrieve stored data. +We'll explore key-based lookups and various pattern matching approaches. ### Key Lookup -Given a key, we can use `lookup/2` to retrieve all records with that key: +The most efficient retrieval method is key lookup using `:ets.lookup/2`: ```elixir iex> :ets.lookup(:user_lookup, "doomspork") [{"doomspork", "Sean", ["Elixir", "Ruby", "Java"]}] ``` -### Simple Matches +For tables that might have multiple objects per key (bag types), this returns all matching objects: -ETS was built for Erlang, so be warned that match variables may feel a _little_ clunky. +```elixir +iex> bag_table = :ets.new(:skills, [:bag, :named_table]) +iex> :ets.insert(:skills, [{"doomspork", "Elixir"}, {"doomspork", "Ruby"}]) +iex> :ets.lookup(:skills, "doomspork") +[{"doomspork", "Elixir"}, {"doomspork", "Ruby"}] +``` -To specify a variable in our match we use the atoms `:"$1"`, `:"$2"`, `:"$3"`, and so on. -The variable number reflects the result position and not the match position. -For values we're not interested in, we use the `:_` variable. +### Simple Pattern Matching -Values can also be used in matching, but only variables will be returned as part of our result. -Let's put it all together and see how it works: +ETS supports pattern matching using special variables. Variables are specified with atoms like `:"$1"`, `:"$2"`, `:"$3"`, and so on. +The variable number reflects the result position, not the match position. +For values we don't care about, use the `:_` variable. + +Let's see how it works: ```elixir iex> :ets.match(:user_lookup, {:"$1", "Sean", :_}) [["doomspork"]] ``` -Let's look at another example to see how variables influence the resulting list order: +Here's how variables influence the result order: ```elixir iex> :ets.match(:user_lookup, {:"$99", :"$1", :"$3"}) @@ -131,7 +220,7 @@ iex> :ets.match(:user_lookup, {:"$99", :"$1", :"$3"}) ["", ["Elixir", "Ruby", "JavaScript"], "3100"]] ``` -What if we want our original object, not a list? We can use `match_object/2`, which regardless of variables returns our entire object: +If we want the complete object, use `:ets.match_object/2`: ```elixir iex> :ets.match_object(:user_lookup, {:"$1", :_, :"$3"}) @@ -142,83 +231,115 @@ iex> :ets.match_object(:user_lookup, {:_, "Sean", :_}) [{"doomspork", "Sean", ["Elixir", "Ruby", "Java"]}] ``` -### Advanced Lookup +### Advanced Querying with Select -We learned about simple match cases but what if we want something more akin to an SQL query? Thankfully there is a more robust syntax available to us. -To lookup our data with `select/2` we need to construct a list of tuples with arity 3. -These tuples represent our pattern, zero or more guards, and a return value format. +For more complex queries, use `:ets.select/2`. This function takes a list of match specifications - tuples with three elements: pattern, guards, and result format. -Our match variables and two new variables, `:"$$"` and `:"$_"`, can be used to construct the return value. -These new variables are shortcuts for the result format; `:"$$"` gets results as lists and `:"$_"` gets the original data objects. +The special variables `:"$$"` and `:"$_"` are shortcuts for result formatting. The `:"$$"` variable returns results as lists while `:"$_"` returns the original data objects. -Let's take one of our previous `match/2` examples and turn it into a `select/2`: +Let's convert a `:ets.match_object/2` example to `:ets.select/2`: ```elixir -iex> :ets.match_object(:user_lookup, {:"$1", :_, :"$3"}) -[{"doomspork", "Sean", ["Elixir", "Ruby", "Java"]}, - {"3100", "", ["Elixir", "Ruby", "JavaScript"]}] - iex> :ets.select(:user_lookup, [{{:"$1", :_, :"$3"}, [], [:"$_"]}]) [{"doomspork", "Sean", ["Elixir", "Ruby", "Java"]}, {"3100", "", ["Elixir", "Ruby", "JavaScript"]}] ``` -Although `select/2` allows for finer control over what and how we retrieve records, the syntax is quite unfriendly and will only become more so. -To handle this the ETS module includes `fun2ms/1`, which turns the functions into match_specs. -With `fun2ms/1` we can create queries using a familiar function syntax. - -Let's use `fun2ms/1` and `select/2` to find all usernames with more than 2 languages: +ETS includes `:ets.fun2ms/1` to convert functions into match specifications for more readable queries: ```elixir iex> fun = :ets.fun2ms(fn {username, _, langs} when length(langs) > 2 -> username end) -{% raw %}[{{:"$1", :_, :"$2"}, [{:>, {:length, :"$2"}, 2}], [:"$1"]}]{% endraw %} +[{{:"$1", :_, :"$2"}, [{:>, {:length, :"$2"}, 2}], [:"$1"]}] iex> :ets.select(:user_lookup, fun) ["doomspork", "3100"] ``` -Want to learn more about the match specification? Check out the official Erlang documentation for [match_spec](http://www.erlang.org/doc/apps/erts/match_spec.html). +### Table Traversal + +ETS provides functions to iterate through tables: + +```elixir +iex> :ets.first(:user_lookup) +"3100" +iex> :ets.next(:user_lookup, "3100") +"doomspork" +``` + +**New in OTP 27**: The `first_lookup/1` and `next_lookup/2` functions combine traversal with lookup: + +```elixir +iex> {key, object} = :ets.first_lookup(:user_lookup) +{"3100", {"3100", "", ["Elixir", "Ruby", "JavaScript"]}} +iex> {next_key, next_object} = :ets.next_lookup(:user_lookup, key) +{"doomspork", {"doomspork", "Sean", ["Elixir", "Ruby", "Java"]}} +``` + +## Updating Data + +### Updating Elements + +`:ets.update_element/4` allows providing a default object when the key doesn't exist: + +```elixir +iex> table = :ets.new(:example, []) +iex> :ets.update_element(table, :key, {2, :new_value}, {:key, :new_value}) +true +iex> :ets.lookup(table, :key) +[{:key, :new_value}] +``` + +### Taking Data + +The `:ets.take/2` function works like `:ets.delete/2` but also returns the deleted objects: + +```elixir +iex> :ets.take(:user_lookup, "doomspork") +[{"doomspork", "Sean", ["Elixir", "Ruby", "Java"]}] +iex> :ets.lookup(:user_lookup, "doomspork") +[] +``` ## Deleting Data ### Removing Records -Deleting terms is as straightforward as `insert/2` and `lookup/2`. -With `delete/2` we only need our table and the key. -This deletes both the key and its values: +Deleting individual records is straightforward with `:ets.delete/2`: ```elixir -iex> :ets.delete(:user_lookup, "doomspork") +iex> :ets.delete(:user_lookup, "3100") true ``` ### Removing Tables -ETS tables are not garbage collected unless the parent is terminated. -Sometimes it may be necessary to delete an entire table without terminating the owner process. -For this we can use `delete/1`: +ETS tables are not garbage collected unless the parent process terminates. +To delete an entire table explicitly, use `:ets.delete/1`: ```elixir iex> :ets.delete(:user_lookup) true ``` -## Example ETS Usage +## Practical Example: Building a Simple Cache -Given what we've learned above, let's put everything together and build a simple cache for expensive operations. -We'll implement a `get/4` function to take a module, function, arguments, and options. -For now the only option we'll worry about is `:ttl`. - -For this example we're assuming the ETS table has been created as part of another process, such as a supervisor: +Let's implement a simple cache for expensive operations using what we've learned: ```elixir defmodule SimpleCache do @moduledoc """ - A simple ETS based cache for expensive function calls. + A simple ETS-based cache for expensive function calls. """ + @table_name :simple_cache + + def init do + :ets.new(@table_name, [:set, :public, :named_table, {:read_concurrency, true}]) + :ok + end + @doc """ - Retrieve a cached value or apply the given function caching and returning + Retrieve a cached value or apply the given function, caching and returning the result. """ def get(mod, fun, args, opts \\ []) do @@ -233,11 +354,11 @@ defmodule SimpleCache do end @doc """ - Lookup a cached result and check the freshness + Lookup a cached result and check if it's still fresh """ defp lookup(mod, fun, args) do - case :ets.lookup(:simple_cache, [mod, fun, args]) do - [result | _] -> check_freshness(result) + case :ets.lookup(@table_name, {mod, fun, args}) do + [{_key, result, expiration}] -> check_freshness(result, expiration) [] -> nil end end @@ -245,10 +366,11 @@ defmodule SimpleCache do @doc """ Compare the result expiration against the current system time. """ - defp check_freshness({mfa, result, expiration}) do - cond do - expiration > :os.system_time(:seconds) -> result - :else -> nil + defp check_freshness(result, expiration) do + if expiration > :os.system_time(:seconds) do + result + else + nil end end @@ -258,69 +380,100 @@ defmodule SimpleCache do defp cache_apply(mod, fun, args, ttl) do result = apply(mod, fun, args) expiration = :os.system_time(:seconds) + ttl - :ets.insert(:simple_cache, {[mod, fun, args], result, expiration}) + :ets.insert(@table_name, {{mod, fun, args}, result, expiration}) result end + + @doc """ + Clear expired entries from the cache + """ + def cleanup do + current_time = :os.system_time(:seconds) + expired_pattern = {{:_, :_, :_}, :_, :"$1"} + guard = [{:<, :"$1", current_time}] + :ets.select_delete(@table_name, [{expired_pattern, guard, [true]}]) + end end ``` -To demonstrate the cache we'll use a function that returns the system time and a TTL of 10 seconds. -As you'll see in the example below, we get the cached result until the value has expired: +Let's test our cache: ```elixir -defmodule ExampleApp do - def test do - :os.system_time(:seconds) - end -end +iex> SimpleCache.init() +:ok + +iex> defmodule ExampleApp do +...> def expensive_operation do +...> :timer.sleep(1000) # Simulate expensive work +...> :os.system_time(:seconds) +...> end +...> end + +iex> SimpleCache.get(ExampleApp, :expensive_operation, [], ttl: 10) +1640995200 # Takes ~1 second -iex> :ets.new(:simple_cache, [:named_table]) -:simple_cache -iex> ExampleApp.test -1451089115 -iex> SimpleCache.get(ExampleApp, :test, [], ttl: 10) -1451089119 -iex> ExampleApp.test -1451089123 -iex> ExampleApp.test -1451089127 -iex> SimpleCache.get(ExampleApp, :test, [], ttl: 10) -1451089119 +iex> SimpleCache.get(ExampleApp, :expensive_operation, [], ttl: 10) +1640995200 # Returns immediately from cache + +# After 10 seconds... +iex> SimpleCache.get(ExampleApp, :expensive_operation, [], ttl: 10) +1640995211 # Takes ~1 second again as cache expired ``` -After 10 seconds if we try again we should get a fresh result: +## Information and Monitoring + +ETS provides several functions to inspect table properties: ```elixir -iex> ExampleApp.test -1451089131 -iex> SimpleCache.get(ExampleApp, :test, [], ttl: 10) -1451089134 +iex> :ets.info(:simple_cache) +[ + {:id, :simple_cache}, + {:decentralized_counters, false}, + {:read_concurrency, true}, + {:write_concurrency, false}, + {:compressed, false}, + {:memory, 305}, + {:owner, #PID<0.123.0>}, + {:heir, :none}, + {:name, :simple_cache}, + {:size, 1}, + {:node, :nonode@nohost}, + {:named_table, true}, + {:type, :set}, + {:keypos, 1}, + {:protection, :public} +] + +iex> :ets.info(:simple_cache, :size) +1 +iex> :ets.info(:simple_cache, :memory) +305 ``` -As you see we are able to implement a scalable and fast cache without any external dependencies and this is only one of many uses for ETS. +## Disk-based Storage with DETS -## Disk-based ETS - -We now know ETS is for in-memory term storage but what if we need disk-based storage? For that we have Disk Based Term Storage, or DETS for short. -The ETS and DETS APIs are interchangeable with the exception of how tables are created. -DETS relies on `open_file/2` and doesn't require the `:named_table` option: +For persistent storage, Erlang provides DETS (Disk-based Term Storage). +The DETS API is nearly identical to ETS, with the main difference being table creation: ```elixir iex> {:ok, table} = :dets.open_file(:disk_storage, [type: :set]) {:ok, :disk_storage} -iex> :dets.insert_new(table, {"doomspork", "Sean", ["Elixir", "Ruby", "Java"]}) -true -iex> select_all = :ets.fun2ms(&(&1)) -[{:"$1", [], [:"$1"]}] -iex> :dets.select(table, select_all) +iex> :dets.insert(table, {"doomspork", "Sean", ["Elixir", "Ruby", "Java"]}) +:ok +iex> :dets.lookup(table, "doomspork") [{"doomspork", "Sean", ["Elixir", "Ruby", "Java"]}] +iex> :dets.close(table) +:ok ``` -If you exit `iex` and look in your local directory, you'll see a new file `disk_storage`: +**Note**: DETS doesn't support `ordered_set` - only `set`, `bag`, and `duplicate_bag`. -```shell -$ ls | grep -c disk_storage -1 -``` +After closing IEx, we'll find a `disk_storage` file in our directory containing the persisted data. + +## Conclusion + +ETS is a powerful in-memory storage solution that provides fast, concurrent access to structured data. +Its flexibility in table types, access controls, and performance optimizations makes it suitable for a wide range of use cases from simple caches to complex data structures. -One last thing to note is that DETS does not support `ordered_set` like ETS, only `set`, `bag`, and `duplicate_bag`. +Combined with DETS for persistence, ETS forms a complete storage solution that can handle everything from temporary caches to application state management. +Understanding ETS is essential for building high-performance Elixir applications that need efficient data access patterns. \ No newline at end of file diff --git a/lessons/en/storage/mnesia.md b/lessons/en/storage/mnesia.md index f2001d4008..ed8acef085 100644 --- a/lessons/en/storage/mnesia.md +++ b/lessons/en/storage/mnesia.md @@ -1,342 +1,696 @@ %{ - version: "1.2.0", + version: "2.0.0", title: "Mnesia", excerpt: """ - Mnesia is a heavy duty real-time distributed database management system. + Mnesia is a distributed real-time database management system that ships with the Erlang Runtime System. In this lesson, we'll explore how to use Mnesia to build fault-tolerant, distributed applications with Elixir. """ } --- ## Overview -Mnesia is a Database Management System (DBMS) that ships with the Erlang Runtime System which we can use naturally with Elixir. -The Mnesia *relational and object hybrid data model* is what makes it suitable for developing distributed applications of any scale. +Mnesia is a Database Management System (DBMS) that comes built into the Erlang Runtime System, making it immediately available in Elixir applications. What makes Mnesia special is its distributed, real-time capabilities and its hybrid data model that combines relational and object-oriented features. -## When to use +Unlike traditional databases that require separate installation and configuration, Mnesia runs inside our Elixir application's virtual machine. This unique architecture provides distributed storage across multiple nodes, real-time performance with flexible in-memory and disk storage options, and fault tolerance through automatic replication and recovery. Mnesia also supports full ACID transactions with rollback capabilities and enables hot code swapping without downtime, making it particularly well-suited for high-availability systems. -When to use a particular piece of technology is often a confusing pursuit. -If you can answer 'yes' to any of the following questions, then this is a good indication to use Mnesia over ETS or DETS. +## When to Use Mnesia -- Do I need to roll back transactions? -- Do I need an easy to use syntax for reading and writing data? -- Should I store data across multiple nodes, rather than one? -- Do I need a choice where to store information (RAM or disk)? +Mnesia shines in specific scenarios. Consider using Mnesia when we can answer "yes" to any of these questions: -## Schema +Do we need a database that's embedded within our application? Are we building a distributed system that needs to share data across multiple nodes? Do we need real-time performance with sub-millisecond response times? Does our application require high availability with automatic failover? Do we need to store and query Elixir data structures directly? -As Mnesia is part of the Erlang core, rather than Elixir, we have to access it with the colon syntax (See Lesson: [Erlang Interoperability](/en/lessons/intermediate/erlang)): +Mnesia might not be the best choice for applications requiring SQL compatibility, systems that need to share data with non-Erlang/Elixir applications, applications with massive datasets that don't fit in memory, or projects that require extensive reporting and analytics. + +## Getting Started + +Since Mnesia is part of the Erlang Runtime System, we access it using the colon syntax for Erlang interoperability but we can give it a more familiar feel if we start by aliasing it for convenience: ```elixir +iex> alias :mnesia, as: Mnesia +``` -iex> :mnesia.create_schema([node()]) +We'll use this approach throughout this lesson to make our code more readable and familiar. -# or if you prefer the Elixir feel... +### Creating a Schema -iex> alias :mnesia, as: Mnesia +Before we can use Mnesia, we need to create a schema. The schema defines the database structure and tells Mnesia where to store data: + +```elixir iex> Mnesia.create_schema([node()]) +:ok ``` -For this lesson, we will take the latter approach when working with the Mnesia API. -`Mnesia.create_schema/1` initializes a new empty schema and passes in a Node List. -In this case, we are passing in the node associated with our IEx session. - -## Nodes +This command creates a new schema on the current node. After running this, you'll notice a new directory in your current working directory named something like `Mnesia.nonode@nohost` - this is where Mnesia stores its data files. -Once we run the `Mnesia.create_schema([node()])` command via IEx, you should see a folder called **Mnesia.nonode@nohost** or similar in your present working directory. -You may be wondering what the **nonode@nohost** means as we haven't come across this before. -Let's have a look. +### Understanding Nodes -```shell -$ iex --help -Usage: iex [options] [.exs file] [data] - - -v Prints version - -e "command" Evaluates the given command (*) - -r "file" Requires the given files/patterns (*) - -S "script"   Finds and executes the given script - -pr "file" Requires the given files/patterns in parallel (*) - -pa "path" Prepends the given path to Erlang code path (*) - -pz "path" Appends the given path to Erlang code path (*) - --app "app" Start the given app and its dependencies (*) - --erl "switches" Switches to be passed down to Erlang (*) - --name "name" Makes and assigns a name to the distributed node - --sname "name" Makes and assigns a short name to the distributed node - --cookie "cookie" Sets a cookie for this distributed node - --hidden Makes a hidden node - --werl Uses Erlang's Windows shell GUI (Windows only) - --detached Starts the Erlang VM detached from console - --remsh "name" Connects to a node using a remote shell - --dot-iex "path" Overrides default .iex.exs file and uses path instead; - path can be empty, then no file will be loaded - -** Options marked with (*) can be given more than once -** Options given after the .exs file or -- are passed down to the executed code -** Options can be passed to the VM using ELIXIR_ERL_OPTIONS or --erl -``` - -When we pass in the `--help` option to IEx from the command line we are presented with all the possible options. -We can see that there is a `--name` and `--sname` options for assigning information to nodes. -A node is just a running Erlang Virtual Machine which handles it's own communications, garbage collection, processing scheduling, memory and more. -The node is being named as **nonode@nohost** simply by default. +The `node()` function returns the current node name. In a standalone IEx session, this might be something like `:nonode@nohost`. In distributed systems, you'll work with named nodes: ```shell -$ iex --name learner@elixirschool.com - -Erlang/OTP {{ site.erlang.OTP }} [erts-{{ site.erlang.erts }}] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] [dtrace] +$ iex --name myapp@localhost +``` -Interactive Elixir ({{ site.elixir.version }}) - press Ctrl+C to exit (type h() ENTER for help) -iex(learner@elixirschool.com)> Node.self -:"learner@elixirschool.com" +```elixir +iex(myapp@localhost)> node() +:myapp@localhost ``` -As we can now see, the node we are running is an atom called `:"learner@elixirschool.com"`. -If we run `Mnesia.create_schema([node()])` again, we will see that it created another folder called **Mnesia.learner@elixirschool.com**. -The purpose of this is quite simple. -Nodes in Erlang are used to connect to other nodes to share (distribute) information and resources. -This doesn't have to be restricted to the same machine and can communicate via LAN, the internet etc. +Node names are important in Mnesia because they determine how data is distributed across your cluster. ## Starting Mnesia -Now we have the background basics out of the way and set up the database, we are now in a position to start the Mnesia DBMS with the `Mnesia.start/0` command. +Once we have a schema, we can start the Mnesia database: ```elixir -iex> alias :mnesia, as: Mnesia -iex> Mnesia.create_schema([node()]) -:ok iex> Mnesia.start() :ok ``` -The function `Mnesia.start/0` is asynchronous. -It starts the initialization of the existing tables and returns the `:ok` atom. -In case we need to perform some actions on an existing table right after starting Mnesia, we need to call the `Mnesia.wait_for_tables/2` function. -It will suspend the caller until the tables are initialized. -See the example in the section [Data initialization and migration](#data-initialization-and-migration). +Mnesia starts asynchronously, so if you need to ensure tables are ready before proceeding, use `wait_for_tables/2`: -It is worth keeping in mind when running a distributed system with two or more participating nodes, the function `Mnesia.start/1` must be executed on all participating nodes. +```elixir +iex> Mnesia.wait_for_tables([:my_table], 5000) +:ok +``` -## Creating Tables +## Working with Tables -The function `Mnesia.create_table/2` is used to create tables within our database. -Below we create a table called `Person` and then pass a keyword list defining the table schema. +Tables in Mnesia are containers for records. Let's create a simple table to store user information: ```elixir -iex> Mnesia.create_table(Person, [attributes: [:id, :name, :job]]) +iex> Mnesia.create_table(:users, [ +...> attributes: [:id, :name, :email, :created_at] +...> ]) {:atomic, :ok} ``` -We define the columns using the atoms `:id`, `:name`, and `:job`. -The first atom (in this case `:id`) is the primary key. -At least one additional attribute is required. +The `attributes` list defines the structure of records in this table. The first attribute (`:id` in this case) automatically becomes the primary key. + +### Table Configuration Options + +Mnesia tables can be configured with various options: + +```elixir +iex> Mnesia.create_table(:sessions, [ +...> attributes: [:session_id, :user_id, :data, :expires_at], +...> type: :set, # Default: each key appears once +...> disc_copies: [node()], # Store on disk and in memory +...> index: [:user_id] # Create secondary index +...> ]) +{:atomic, :ok} +``` -When we execute `Mnesia.create_table/2`, it will return either one of the following responses: +**Table types:** +`:set` means each key appears once (default), `:ordered_set` provides ordered keys useful for range queries, `:bag` allows multiple values per key but no duplicates, and `:duplicate_bag` permits multiple values per key including duplicates. -- `{:atomic, :ok}` if the function executes successfully -- `{:aborted, Reason}` if the function failed +**Storage types:** +`:ram_copies` stores data in memory only for fastest access but data is lost on restart, `:disc_copies` stores data in both memory and disk providing a good balance of performance and durability, and `:disc_only_copies` stores data on disk only using minimal memory but with slower access. -In particular, if the table already exists, the reason will be of the form `{:already_exists, table}` so if we try to create this table a second time, we will get: +## Basic Operations + +### Writing Data + +Let's add some users to our table. Mnesia stores data as tuples where the first element is the table name: ```elixir -iex> Mnesia.create_table(Person, [attributes: [:id, :name, :job]]) -{:aborted, {:already_exists, Person}} +iex> user1 = {:users, 1, "Alice", "alice@example.com", ~N[2024-01-01 10:00:00]} +iex> user2 = {:users, 2, "Bob", "bob@example.com", ~N[2024-01-01 11:00:00]} + +iex> Mnesia.transaction(fn -> +...> Mnesia.write(user1) +...> Mnesia.write(user2) +...> end) +{:atomic, :ok} ``` -## The Dirty Way +All Mnesia operations that modify data should be wrapped in transactions to ensure consistency. + +### Reading Data -First of all we will look at the dirty way of reading and writing to a Mnesia table. -This should generally be avoided as success is not guaranteed, but it should help us learn and become comfortable working with Mnesia. -Let's add some entries to our **Person** table. +Reading data is straightforward: ```elixir -iex> Mnesia.dirty_write({Person, 1, "Seymour Skinner", "Principal"}) -:ok +iex> Mnesia.transaction(fn -> +...> Mnesia.read(:users, 1) +...> end) +{:atomic, [{:users, 1, "Alice", "alice@example.com", ~N[2024-01-01 10:00:00]}]} +``` + +If no record is found, you'll get an empty list: + +```elixir +iex> Mnesia.transaction(fn -> +...> Mnesia.read(:users, 999) +...> end) +{:atomic, []} +``` + +### Updating Data + +To update a record, simply write a new version with the same key: + +```elixir +iex> updated_user = {:users, 1, "Alice Smith", "alice.smith@example.com", ~N[2024-01-01 10:00:00]} +iex> Mnesia.transaction(fn -> +...> Mnesia.write(updated_user) +...> end) +{:atomic, :ok} +``` + +### Deleting Data + +Delete records by specifying the table and key: + +```elixir +iex> Mnesia.transaction(fn -> +...> Mnesia.delete({:users, 2}) +...> end) +{:atomic, :ok} +``` + +## Dirty Operations -iex> Mnesia.dirty_write({Person, 2, "Homer Simpson", "Safety Inspector"}) +For performance-critical operations where you don't need transaction guarantees, Mnesia provides "dirty" operations: + +```elixir +iex> user3 = {:users, 3, "Charlie", "charlie@example.com", ~N[2024-01-01 12:00:00]} +iex> Mnesia.dirty_write(user3) :ok -iex> Mnesia.dirty_write({Person, 3, "Moe Szyslak", "Bartender"}) +iex> Mnesia.dirty_read(:users, 3) +[{:users, 3, "Charlie", "charlie@example.com", ~N[2024-01-01 12:00:00]}] + +iex> Mnesia.dirty_delete({:users, 3}) :ok ``` -...and to retrieve the entries we can use `Mnesia.dirty_read/1`: +**Warning:** Dirty operations bypass transaction safety. Use them only when we understand the implications and need maximum performance. + +## Querying Data + +Mnesia provides several ways to query data beyond simple key lookups. + +### Using Indices + +Remember the index we created on `:user_id` for the sessions table? Let's use it: ```elixir -iex> Mnesia.dirty_read({Person, 1}) -[{Person, 1, "Seymour Skinner", "Principal"}] +iex> Mnesia.create_table(:sessions, [ +...> attributes: [:session_id, :user_id, :data, :expires_at], +...> index: [:user_id] +...> ]) +{:atomic, :ok} -iex> Mnesia.dirty_read({Person, 2}) -[{Person, 2, "Homer Simpson", "Safety Inspector"}] +iex> sessions = [ +...> {:sessions, "sess_1", 1, %{theme: "dark"}, ~N[2024-01-02 10:00:00]}, +...> {:sessions, "sess_2", 1, %{theme: "light"}, ~N[2024-01-02 11:00:00]}, +...> {:sessions, "sess_3", 2, %{theme: "auto"}, ~N[2024-01-02 12:00:00]} +...> ] -iex> Mnesia.dirty_read({Person, 3}) -[{Person, 3, "Moe Szyslak", "Bartender"}] +iex> Mnesia.transaction(fn -> +...> Enum.each(sessions, &Mnesia.write/1) +...> end) +{:atomic, :ok} -iex> Mnesia.dirty_read({Person, 4}) -[] +iex> Mnesia.transaction(fn -> +...> Mnesia.index_read(:sessions, 1, :user_id) +...> end) +{:atomic, [ + {:sessions, "sess_1", 1, %{theme: "dark"}, ~N[2024-01-02 10:00:00]}, + {:sessions, "sess_2", 1, %{theme: "light"}, ~N[2024-01-02 11:00:00]} +]} ``` -If we try to query a record that doesn't exist Mnesia will respond with an empty list. +### Pattern Matching + +Mnesia supports pattern matching with the special atom `:_` as a wildcard: + +```elixir +iex> Mnesia.transaction(fn -> +...> Mnesia.match_object({:users, :_, :_, "alice@example.com", :_}) +...> end) +{:atomic, [{:users, 1, "Alice", "alice@example.com", ~N[2024-01-01 10:00:00]}]} +``` -## Transactions +### Advanced Queries with select/2 -Traditionally we use **transactions** to encapsulate our reads and writes to our database. -Transactions are an important part of designing fault-tolerant, highly distributed systems. -An Mnesia *transaction is a mechanism by which a series of database operations can be executed as one functional block*. -First we create an anonymous function, in this case `data_to_write` and then pass it onto `Mnesia.transaction`. +For complex queries, use `select/2` with match specifications: ```elixir -iex> data_to_write = fn -> -...> Mnesia.write({Person, 4, "Marge Simpson", "home maker"}) -...> Mnesia.write({Person, 5, "Hans Moleman", "unknown"}) -...> Mnesia.write({Person, 6, "Monty Burns", "Businessman"}) -...> Mnesia.write({Person, 7, "Waylon Smithers", "Executive assistant"}) -...> end -#Function<20.54118792/0 in :erl_eval.expr/5> +iex> Mnesia.transaction(fn -> +...> # Find all users whose names start with "A" +...> Mnesia.select(:users, [ +...> { +...> {:users, :"$1", :"$2", :"$3", :"$4"}, # Match pattern +...> [{:==, {:binary_part, :"$2", 0, 1}, <<"A">>}], # Guard (name starts with "A") +...> [:"$$"] # Return entire match +...> } +...> ]) +...> end) +{:atomic, [[1, "Alice", "alice@example.com", ~N[2024-01-01 10:00:00]]]} +``` -iex> Mnesia.transaction(data_to_write) -{:atomic, :ok} +## Error Handling and Recovery + +Mnesia operations return specific patterns that make error handling straightforward: + +```elixir +iex> case Mnesia.transaction(fn -> Mnesia.read(:users, 1) end) do +...> {:atomic, []} -> +...> IO.puts("User not found") +...> {:atomic, [user]} -> +...> IO.puts("Found user: #{inspect(user)}") +...> {:aborted, reason} -> +...> IO.puts("Transaction failed: #{inspect(reason)}") +...> end +Found user: {:users, 1, "Alice", "alice@example.com", ~N[2024-01-01 10:00:00]} ``` -Based on this transaction message, we can safely assume that we have written the data to our `Person` table. -Let's use a transaction to read from the database now to make sure. -We will use `Mnesia.read/1` to read from the database, but again from within an anonymous function. +### Handling Table Creation Errors ```elixir -iex> data_to_read = fn -> -...> Mnesia.read({Person, 6}) +iex> case Mnesia.create_table(:users, [attributes: [:id, :name]]) do +...> {:atomic, :ok} -> +...> IO.puts("Table created successfully") +...> {:aborted, {:already_exists, :users}} -> +...> IO.puts("Table already exists") +...> {:aborted, reason} -> +...> IO.puts("Failed to create table: #{inspect(reason)}") ...> end -#Function<20.54118792/0 in :erl_eval.expr/5> +Table already exists +``` + +## Building a Real Application -iex> Mnesia.transaction(data_to_read) -{:atomic, [{Person, 6, "Monty Burns", "Businessman"}]} +Let's build a simple blog system to demonstrate Mnesia in action: + +```elixir +defmodule Blog.Database do + alias :mnesia, as: Mnesia + + def setup do + # Create schema if it doesn't exist + case Mnesia.create_schema([node()]) do + :ok -> :ok + {:error, {_, {:already_exists, _}}} -> :ok + {:error, reason} -> {:error, {:schema_creation_failed, reason}} + end + + # Start Mnesia + :ok = Mnesia.start() + + # Create tables + create_tables() + end + + defp create_tables do + # Posts table + Mnesia.create_table(:posts, [ + attributes: [:id, :title, :content, :author_id, :published_at, :tags], + index: [:author_id] + ]) + + # Authors table + Mnesia.create_table(:authors, [ + attributes: [:id, :name, :email, :bio] + ]) + + # Comments table + Mnesia.create_table(:comments, [ + attributes: [:id, :post_id, :author_name, :content, :created_at], + index: [:post_id], + type: :bag # Multiple comments per post + ]) + end + + def create_author(name, email, bio \\ "") do + id = :erlang.unique_integer([:positive]) + author = {:authors, id, name, email, bio} + + case Mnesia.transaction(fn -> Mnesia.write(author) end) do + {:atomic, :ok} -> {:ok, id} + {:aborted, reason} -> {:error, reason} + end + end + + def create_post(title, content, author_id, tags \\ []) do + id = :erlang.unique_integer([:positive]) + post = {:posts, id, title, content, author_id, DateTime.utc_now(), tags} + + case Mnesia.transaction(fn -> Mnesia.write(post) end) do + {:atomic, :ok} -> {:ok, id} + {:aborted, reason} -> {:error, reason} + end + end + + def get_posts_by_author(author_id) do + case Mnesia.transaction(fn -> + Mnesia.index_read(:posts, author_id, :author_id) + end) do + {:atomic, posts} -> {:ok, posts} + {:aborted, reason} -> {:error, reason} + end + end + + def add_comment(post_id, author_name, content) do + id = :erlang.unique_integer([:positive]) + comment = {:comments, id, post_id, author_name, content, DateTime.utc_now()} + + case Mnesia.transaction(fn -> Mnesia.write(comment) end) do + {:atomic, :ok} -> {:ok, id} + {:aborted, reason} -> {:error, reason} + end + end + + def get_comments_for_post(post_id) do + case Mnesia.transaction(fn -> + Mnesia.index_read(:comments, post_id, :post_id) + end) do + {:atomic, comments} -> {:ok, comments} + {:aborted, reason} -> {:error, reason} + end + end +end ``` -Note that if you want to update data, you just need to call `Mnesia.write/1` with the same key as an existing record. -Therefore, to update the record for Hans, you can do: +Let's test our blog system: ```elixir -iex> Mnesia.transaction( -...> fn -> -...> Mnesia.write({Person, 5, "Hans Moleman", "Ex-Mayor"}) -...> end +iex> Blog.Database.setup() +:ok + +iex> {:ok, author_id} = Blog.Database.create_author("Jane Doe", "jane@example.com", "Tech writer") +{:ok, 123456} + +iex> {:ok, post_id} = Blog.Database.create_post( +...> "Getting Started with Mnesia", +...> "Mnesia is a powerful database...", +...> author_id, +...> ["elixir", "database"] ...> ) +{:ok, 234567} + +iex> Blog.Database.add_comment(post_id, "Bob Reader", "Great post!") +{:ok, 345678} + +iex> {:ok, comments} = Blog.Database.get_comments_for_post(post_id) +{:ok, [{:comments, 345678, 234567, "Bob Reader", "Great post!", ~U[2024-01-01 15:30:00.123456Z]}]} +``` + +## Distribution and Replication + +One of Mnesia's greatest strengths is its built-in support for distribution. Here's how to set up a distributed Mnesia cluster: + +### Setting Up Multiple Nodes + +Start two named nodes in separate terminals: + +```shell +# Terminal 1 +$ iex --name node1@localhost --cookie mycookie + +# Terminal 2 +$ iex --name node2@localhost --cookie mycookie +``` + +### Creating a Distributed Schema + +On the first node: + +```elixir +# node1@localhost +iex> alias :mnesia, as: Mnesia +iex> Mnesia.create_schema([node(), :node2@localhost]) +:ok ``` -## Using indices +This creates a schema that will be shared across both nodes. + +### Starting Mnesia on All Nodes -Mnesia support indices on non-key columns and data can then be queried against those indices. -So we can add an index against the `:job` column of the `Person` table: +Start Mnesia on both nodes: ```elixir -iex> Mnesia.add_table_index(Person, :job) +# On both nodes +iex> Mnesia.start() +:ok +``` + +### Creating Replicated Tables + +```elixir +# On node1@localhost +iex> Mnesia.create_table(:distributed_users, [ +...> attributes: [:id, :name, :email], +...> disc_copies: [node(), :node2@localhost] # Replicate to both nodes +...> ]) {:atomic, :ok} ``` -The result is similar to the one returned by `Mnesia.create_table/2`: +Now data written to this table will be automatically replicated to both nodes! -- `{:atomic, :ok}` if the function executes successfully -- `{:aborted, Reason}` if the function failed +### Testing Replication -In particular, if the index already exists, the reason will be of the form `{:already_exists, table, attribute_index}` so if we try to add this index a second time, we will get: +Write data on one node: ```elixir -iex> Mnesia.add_table_index(Person, :job) -{:aborted, {:already_exists, Person, 4}} +# On node1@localhost +iex> Mnesia.transaction(fn -> +...> Mnesia.write({:distributed_users, 1, "Alice", "alice@example.com"}) +...> end) +{:atomic, :ok} ``` -Once the index is successfully created, we can read against it and retrieve a list of all principals: +Read it from the other node: ```elixir -iex> Mnesia.transaction( -...> fn -> -...> Mnesia.index_read(Person, "Principal", :job) -...> end -...> ) -{:atomic, [{Person, 1, "Seymour Skinner", "Principal"}]} +# On node2@localhost +iex> Mnesia.transaction(fn -> +...> Mnesia.read(:distributed_users, 1) +...> end) +{:atomic, [{:distributed_users, 1, "Alice", "alice@example.com"}]} ``` -## Match and select +## Data Migration and Schema Evolution -Mnesia supports complex queries to retrieve data from a table in the form of matching and ad-hoc select functions. +As our application evolves, we'll need to modify our database schema. Mnesia provides tools for this: -The `Mnesia.match_object/1` function returns all records that match the given pattern. -If any of the columns in the table have indices, it can make use of them to make the query more efficient. -Use the special atom `:_` to identify columns that don't participate in the match. +### Adding New Tables ```elixir -iex> Mnesia.transaction( -...> fn -> -...> Mnesia.match_object({Person, :_, "Marge Simpson", :_}) -...> end -...> ) -{:atomic, [{Person, 4, "Marge Simpson", "home maker"}]} +iex> Mnesia.create_table(:user_preferences, [ +...> attributes: [:user_id, :theme, :language, :notifications] +...> ]) +{:atomic, :ok} ``` -The `Mnesia.select/2` function allows you to specify a custom query using any operator or function in the Elixir language (or Erlang for that matter). -Let's look at an example to select all records that have a key that is greater than 3: +### Adding Columns to Existing Tables ```elixir -iex> Mnesia.transaction( -...> fn -> -...> Mnesia.select(Person, [{{Person, :"$1", :"$2", :"$3"}, [{:>, :"$1", 3}], [:"$$"]}]) -...> end +iex> Mnesia.transform_table(:users, +...> fn({:users, id, name, email, created_at}) -> +...> {:users, id, name, email, created_at, true} # Add 'active' field +...> end, +...> [:id, :name, :email, :created_at, :active] ...> ) -{:atomic, [[7, "Waylon Smithers", "Executive assistant"], [4, "Marge Simpson", "home maker"], [6, "Monty Burns", "Businessman"], [5, "Hans Moleman", "unknown"]]} -``` - -Let's unpack this. -The first attribute is the table, `Person`, the second attribute is a tuple of the form `{match, [guard], [result]}`: - -- `match` is the same as what you'd pass to the `Mnesia.match_object/1` function; however, note the special atoms `:"$n"` that specify positional parameters that are used by the remainder of the query -- the `guard` list is a list of tuples that specifies what guard functions to apply, in this case the `:>` (greater than) built in function with the first positional parameter `:"$1"` and the constant `3` as attributes -- the `result` list is the list of fields that are returned by the query, in the form of positional parameters of the special atom `:"$$"` to reference all fields so you could use `[:"$1", :"$2"]` to return the first two fields or `[:"$$"]` to return all fields - -For more details, see [the Erlang Mnesia documentation for select/2](http://erlang.org/doc/man/mnesia.html#select-2). - -## Data initialization and migration - -With every software solution, there will come a time when you need to upgrade the software and migrate the data stored in your database. -For example, we may want to add an `:age` column to our `Person` table in v2 of our app. -We can't create the `Person` table once it's been created but we can transform it. -For this we need to know when to transform, which we can do when creating the table. -To do this, we can use the `Mnesia.table_info/2` function to retrieve the current structure of the table and the `Mnesia.transform_table/3` function to transform it to the new structure. - -The code below does this by implementing the following logic: - -- Create the table with the v2 attributes: `[:id, :name, :job, :age]` -- Handle the creation result: - - `{:atomic, :ok}`: initialize the table by creating indices on `:job` and `:age` - - `{:aborted, {:already_exists, Person}}`: check what the attributes are in the current table and act accordingly: - - if it's the v1 list (`[:id, :name, :job]`), transform the table giving everybody an age of 21 and add a new index on `:age` - - if it's the v2 list, do nothing, we're good - - if it's something else, bail out - -If we are performing any actions on the existing tables right after starting Mnesia with `Mnesia.start/0`, those tables may not be initialized and accessible. -In that case, we should use the [`Mnesia.wait_for_tables/2`](http://erlang.org/doc/man/mnesia.html#wait_for_tables-2) function. -It will suspend the current process until the tables are initialized or until the timeout is reached. - -The `Mnesia.transform_table/3` function takes as attributes the name of the table, a function that transforms a record from the old to the new format and the list of new attributes. - -```elixir -case Mnesia.create_table(Person, [attributes: [:id, :name, :job, :age]]) do - {:atomic, :ok} -> - Mnesia.add_table_index(Person, :job) - Mnesia.add_table_index(Person, :age) - {:aborted, {:already_exists, Person}} -> - case Mnesia.table_info(Person, :attributes) do - [:id, :name, :job] -> - Mnesia.wait_for_tables([Person], 5000) - Mnesia.transform_table( - Person, - fn ({Person, id, name, job}) -> - {Person, id, name, job, 21} - end, - [:id, :name, :job, :age] - ) - Mnesia.add_table_index(Person, :age) - [:id, :name, :job, :age] -> - :ok - other -> - {:error, other} +{:atomic, :ok} +``` + +### Creating Indices on Existing Tables + +```elixir +iex> Mnesia.add_table_index(:users, :email) +{:atomic, :ok} +``` + +## Best Practices + +### Use Transactions for Consistency + +Always use transactions for operations that must be atomic: + +```elixir +# Good: Transfer money between accounts atomically +def transfer_money(from_id, to_id, amount) do + Mnesia.transaction(fn -> + [{:accounts, ^from_id, from_balance}] = Mnesia.read(:accounts, from_id) + [{:accounts, ^to_id, to_balance}] = Mnesia.read(:accounts, to_id) + + if from_balance >= amount do + Mnesia.write({:accounts, from_id, from_balance - amount}) + Mnesia.write({:accounts, to_id, to_balance + amount}) + :ok + else + Mnesia.abort(:insufficient_funds) end + end) +end +``` + +### Design Tables for Our Query Patterns + +Create indices for fields we'll query frequently. If we often search users by email, we should structure our tables accordingly: + +```elixir +# If we often search users by email +Mnesia.create_table(:users, [ + attributes: [:id, :name, :email, :created_at], + index: [:email] +]) +``` + +### Choose Appropriate Storage Types + +Select storage types based on our application's needs. Use `:ram_copies` for cache-like data that doesn't need to survive restarts, `:disc_copies` for important data that needs to survive restarts while maintaining good performance, and `:disc_only_copies` for large datasets that don't fit in memory. + +### Handle Network Partitions + +In distributed systems, we should be prepared for network partitions: + +```elixir +require Logger + +def robust_write(table, record) do + case Mnesia.transaction(fn -> Mnesia.write(record) end) do + {:atomic, :ok} -> :ok + {:aborted, reason} -> + Logger.error("Write failed: #{inspect(reason)}") + {:error, reason} + end +end +``` + +### Monitor Table Sizes + +Keep an eye on table sizes, especially for `:ram_copies` tables: + +```elixir +iex> Mnesia.table_info(:users, :size) +2 + +iex> Mnesia.table_info(:users, :memory) +1024 # in words +``` + +## Performance Considerations + +### Choosing the Right Table Type + +`:set` works best for key-value lookups, `:ordered_set` excels when we need sorted data or range queries, `:bag` handles one-to-many relationships effectively, and `:duplicate_bag` serves cases where duplicates are meaningful. + +### Optimizing Queries + +Use indices for frequently queried fields: + +```elixir +# Slow: Full table scan +Mnesia.match_object({:users, :_, :_, "alice@example.com", :_}) + +# Fast: Index lookup (if email is indexed) +Mnesia.index_read(:users, "alice@example.com", :email) +``` + +### Batch Operations + +Group multiple writes in a single transaction: + +```elixir +# Good: Single transaction for multiple writes +Mnesia.transaction(fn -> + Enum.each(users, &Mnesia.write/1) +end) + +# Less efficient: Multiple transactions +Enum.each(users, fn user -> + Mnesia.transaction(fn -> Mnesia.write(user) end) +end) +``` + +## Debugging and Troubleshooting + +### Inspecting Table Information + +```elixir +# Get all table information +iex> Mnesia.table_info(:users, :all) + +# Get specific information +iex> Mnesia.table_info(:users, :attributes) +[:id, :name, :email, :created_at, :active] + +iex> Mnesia.table_info(:users, :storage_type) +:disc_copies + +iex> Mnesia.table_info(:users, :size) +1 +``` + +### Viewing All Records + +```elixir +iex> Mnesia.transaction(fn -> +...> Mnesia.foldr(fn(record, acc) -> [record | acc] end, [], :users) +...> end) +{:atomic, [{:users, 1, "Alice Smith", "alice.smith@example.com", ~N[2024-01-01 10:00:00], true}]} +``` + +### Checking System Status + +```elixir +iex> Mnesia.system_info(:running_db_nodes) +[:nonode@nohost] + +iex> Mnesia.system_info(:tables) +[:schema, :users, :sessions, :posts, :authors, :comments, :user_preferences, :distributed_users] +``` + +## Common Pitfalls and Solutions + +### Schema Already Exists Error + +```elixir +# Problem: Running create_schema multiple times +iex> Mnesia.create_schema([node()]) +{:error, {:nonode@nohost, {:already_exists, :nonode@nohost}}} + +# Solution: Check if schema exists first +case Mnesia.create_schema([node()]) do + :ok -> :ok + {:error, {_, {:already_exists, _}}} -> :ok + error -> error +end +``` + +### Table Already Exists Error + +```elixir +# Solution: Use conditional creation +def ensure_table_exists(name, options) do + case Mnesia.create_table(name, options) do + {:atomic, :ok} -> :ok + {:aborted, {:already_exists, ^name}} -> :ok + error -> error + end +end +``` + +### Transaction Aborts + +```elixir +# Handle transaction aborts gracefully +case Mnesia.transaction(fn -> + # Our operations here +end) do + {:atomic, result} -> {:ok, result} + {:aborted, :no_transaction} -> {:error, :database_not_available} + {:aborted, reason} -> {:error, reason} end ``` + +## Conclusion + +Mnesia is a powerful, distributed database that's perfect for Elixir applications requiring real-time performance and fault tolerance. Its tight integration with the BEAM virtual machine makes it unique among database solutions. + +Mnesia excels in scenarios requiring embedded databases, real-time performance, and distributed architectures. While it may not be suitable for every use case, it can be the perfect solution for applications that fit its strengths. + +For more advanced topics like hot code swapping, custom backends, and performance tuning, explore the [official Mnesia documentation](http://erlang.org/doc/man/mnesia.html). diff --git a/lessons/en/storage/redix.md b/lessons/en/storage/redix.md new file mode 100644 index 0000000000..27fb5d95bf --- /dev/null +++ b/lessons/en/storage/redix.md @@ -0,0 +1,423 @@ +%{ + version: "1.0.0", + title: "Redix", + excerpt: """ + Redix is a fast, pipelined, and resilient Redis driver for Elixir. In this lesson, we'll explore how to integrate Redis into our Elixir applications using Redix, covering everything from basic operations to advanced patterns like connection pooling and pub/sub messaging. + """ +} +--- + +## What is Redix? + +[Redix](https://github.com/whatyouhide/redix) is the go-to Redis client for Elixir applications, supporting both Redis and Valkey (the Redis fork). It's designed to be fast, resilient, and easy to use while leveraging Redis's pipelining capabilities for optimal performance. Unlike some Redis clients that try to abstract Redis commands, Redix embraces Redis's native command structure, making it both powerful and straightforward. + +Redix supports pipelining for sending multiple commands in a single round-trip, automatic reconnection with configurable backoff strategies, pub/sub for real-time messaging, and Redis Sentinel integration for high availability. It also includes telemetry events out of the box, making monitoring straightforward. + +## Installation + +To get started with Redix, add it to our `mix.exs`: + +```elixir +defp deps do + [ + {:redix, "~> 1.1"} + ] +end +``` + +Then fetch the dependencies: + +```shell +mix deps.get +``` + +## Basic Usage + +### Connecting to Redis + +Let's start by establishing a connection to Redis using `Redix.start_link/1`: + +```elixir +# Connect to Redis on localhost:6379 (default) +{:ok, conn} = Redix.start_link() + +# Connect to a specific host and port +{:ok, conn} = Redix.start_link(host: "example.com", port: 5000) + +# Connect using a Redis URI +{:ok, conn} = Redix.start_link("redis://localhost:6379/3") +``` +### Using Named Connections + +For real-world applications, we generally want to start Redix connections under our application's supervision tree with registered names. Let's pull the Redis URL from our config rather than hardcoding it: + +```elixir +# config/config.exs +config :my_app, redis_url: "redis://localhost:6379" +``` + +```elixir +def start(_type, _args) do + redis_url = Application.get_env(:my_app, :redis_url, "redis://localhost:6379") + + children = [ + {Redix, name: :redix, url: redis_url} + # ...other children + ] + + opts = [strategy: :one_for_one, name: MyApp.Supervisor] + Supervisor.start_link(children, opts) +end +``` + +Now we can use the connection anywhere in our application: + +```elixir +iex> Redix.command(:redix, ["SET", "app_state", "running"]) +{:ok, "OK"} + +iex> Redix.command!(:redix, ["GET", "app_state"]) +"running" +``` + +### Executing Commands + +Redix uses Redis's native command structure represented as lists of strings. This means there's no abstraction layer to learn — if we know a Redis command, we can use it directly. A complete list of commands can be found in the official Redis documentation. + +```elixir +iex> Redix.command(conn, ["SET", "mykey", "Hello, Redis!"]) +{:ok, "OK"} + +iex> Redix.command(conn, ["GET", "mykey"]) +{:ok, "Hello, Redis!"} + +iex> Redix.command(conn, ["INCR", "counter"]) +{:ok, 1} + +iex> Redix.command(conn, ["INCR", "counter"]) +{:ok, 2} +``` + +For cases where we want to work with the result directly or let errors bubble up, Redix provides bang (`!`) variants: + +```elixir +iex> Redix.command!(conn, ["PING"]) +"PONG" + +iex> Redix.command!(conn, ["GET", "mykey"]) +"Hello, Redis!" + +iex> Redix.command!(conn, ["INVALID", "COMMAND"]) +** (Redix.Error) ERR unknown command 'INVALID' +``` + +### Pipelining Commands + +One of Redix's powerful features is command pipelining which allows us to send multiple commands at once. This dramatically improves performance when we need to execute several commands: + +```elixir +iex> commands = [ +...> ["SET", "key1", "value1"], +...> ["SET", "key2", "value2"], +...> ["GET", "key1"], +...> ["GET", "key2"] +...> ] + +iex> Redix.pipeline(conn, commands) +{:ok, ["OK", "OK", "value1", "value2"]} + +iex> Redix.pipeline!(conn, [["INCR", "foo"], ["INCR", "foo"], ["INCRBY", "foo", "2"]]) +[1, 2, 4] +``` + +As we see from our examples, pipeline commands return results in the same order as the commands were sent, making it easy to correlate commands with their responses. + +## Working with Data Types + +Redis supports various data types and Redix makes it easy to work with all of them. + +### Strings + +```elixir +# Set and get strings +iex> Redix.command!(conn, ["SET", "username", "alice"]) +"OK" +iex> Redix.command!(conn, ["GET", "username"]) +"alice" +``` + +### Atomic operations + +```elixir +iex> Redix.command!(conn, ["INCR", "page_views"]) +1 +iex> Redix.command!(conn, ["INCRBY", "page_views", "5"]) +6 +``` + +### Lists + +```elixir +# Push elements to a list +iex> Redix.command!(conn, ["LPUSH", "tasks", "task1"]) +iex> Redix.command!(conn, ["LPUSH", "tasks", "task2"]) +iex> Redix.command!(conn, ["RPUSH", "tasks", "task3"]) + +# Get list contents +iex> Redix.command!(conn, ["LRANGE", "tasks", "0", "-1"]) +["task3", "task2", "task1"] + +# Pop elements +iex> Redix.command!(conn, ["LPOP", "tasks"]) +"task3" +``` + +### Sets + +```elixir +# Add members to a set +iex> Redix.command!(conn, ["SADD", "languages", "elixir"]) +iex> Redix.command!(conn, ["SADD", "languages", "erlang", "go", "rust"]) + +# Get all members +iex> Redix.command!(conn, ["SMEMBERS", "languages"]) +["elixir", "erlang", "go", "rust"] + +# Check membership +iex> Redix.command!(conn, ["SISMEMBER", "languages", "elixir"]) +1 +``` + +### Hashes + +```elixir +# Set hash fields +iex> Redix.command!(conn, ["HSET", "user:1", "name", "Alice", "age", "30"]) + +# Get specific fields +iex> Redix.command!(conn, ["HGET", "user:1", "name"]) +"Alice" + +# Get all fields and values +iex> Redix.command!(conn, ["HGETALL", "user:1"]) +["name", "Alice", "age", "30"] +``` + +## Telemetry Integration + +Redix emits telemetry events that we can use for monitoring. Let's look at an example Telemetry setup for our application: + +```elixir +defmodule MyApp.RedixTelemetry do + require Logger + + def setup do + events = [ + [:redix, :connection], + [:redix, :disconnection], + [:redix, :failed_connection], + [:redix, :pipeline_stop], + [:redix, :command_stop] + ] + + :telemetry.attach_many( + "redix-telemetry", + events, + &handle_event/4, + %{} + ) + end + + def handle_event([:redix, :connection], _measurements, metadata, _config) do + Logger.info("Connected to Redis at #{metadata.address}") + end + + def handle_event([:redix, :disconnection], _measurements, metadata, _config) do + Logger.warn("Disconnected from Redis at #{metadata.address}: #{Exception.message(metadata.reason)}") + end + + def handle_event([:redix, :failed_connection], _measurements, metadata, _config) do + Logger.error("Failed to connect to Redis at #{metadata.address}: #{Exception.message(metadata.reason)}") + end + + def handle_event([:redix, :command_stop], measurements, metadata, _config) do + if measurements.duration > 1_000_000 do # Log slow commands (>1ms) + Logger.warn("Slow Redis command: #{inspect(metadata.command)} took #{measurements.duration}μs") + end + end + + def handle_event([:redix, :pipeline_stop], measurements, metadata, _config) do + command_count = length(metadata.commands) + Logger.debug("Pipeline with #{command_count} commands took #{measurements.duration}μs") + end +end +``` + +We have to remember to start telemetry in our application: + +```elixir +def start(_type, _args) do + MyApp.RedixTelemetry.setup() + + children = [ + {Redix, name: :redix}, + # ...other children + ] + + Supervisor.start_link(children, strategy: :one_for_one) +end +``` + +## Real-World Examples + +### Caching with TTL + +Setting TTLs (Time To Live) on cached data prevents our Redis instance from running out of memory and keeps data fresh by expiring stale entries. Let's look at an example: + +```elixir +defmodule MyApp.Cache do + @redix_name :cache_redix + + def get(key) do + case Redix.command(@redix_name, ["GET", key]) do + {:ok, nil} -> {:error, :not_found} + {:ok, value} -> {:ok, Jason.decode!(value)} + error -> error + end + end + + def set(key, value, ttl \\ 3600) do + json_value = Jason.encode!(value) + Redix.command(@redix_name, ["SETEX", key, ttl, json_value]) + end + + def delete(key) do + Redix.command(@redix_name, ["DEL", key]) + end + + def exists?(key) do + case Redix.command(@redix_name, ["EXISTS", key]) do + {:ok, 1} -> true + _ -> false + end + end +end +``` + +### Rate Limiting + +Rate limiting prevents abuse by restricting how many requests a user or IP can make within a time window. We can use Redis's atomic `INCR` and `EXPIRE` commands in a pipeline to track request counts without race conditions. + +```elixir +defmodule MyApp.RateLimit do + @redix_name :rate_limit_redix + + def check_rate_limit(identifier, limit, window_seconds) do + key = "rate_limit:#{identifier}" + + # Use a Lua script to atomically increment and set expiry only on first request + script = """ + local current = redis.call("INCR", KEYS[1]) + if current == 1 then + redis.call("EXPIRE", KEYS[1], ARGV[1]) + end + return current + """ + + case Redix.command(@redix_name, ["EVAL", script, "1", key, window_seconds]) do + {:ok, count} when count <= limit -> + {:ok, %{allowed: true, count: count, limit: limit}} + + {:ok, count} -> + {:ok, %{allowed: false, count: count, limit: limit}} + + error -> + error + end + end +end + +defmodule MyApp.RateLimitPlug do + import Plug.Conn + + @behaviour Plug + + def init(opts), do: opts + + def call(conn, opts) do + identifier = get_identifier(conn, opts) + limit = Keyword.fetch!(opts, :limit) + window = Keyword.fetch!(opts, :window) + + case MyApp.RateLimit.check_rate_limit(identifier, limit, window) do + {:ok, %{allowed: true}} -> + conn + + {:ok, %{allowed: false, count: count, limit: limit}} -> + conn + |> send_resp(429, "Rate limit exceeded: #{count}/#{limit}") + |> halt() + + {:error, reason} -> + # In case Redix/Redis is down, allow request + conn + end + end + + # Extract identifier (by default, IP) + defp get_identifier(conn, opts) do + case Keyword.get(opts, :identifier) do + :ip -> + to_string(:inet_parse.ntoa(conn.remote_ip)) + {:header, name} -> + get_req_header(conn, name) |> List.first() || "anonymous" + val when is_function(val, 1) -> + val.(conn) + nil -> + # Default fallback: IP + to_string(:inet_parse.ntoa(conn.remote_ip)) + end + end +end +``` + +## Transactions + +When we need atomicity across multiple commands, Redis provides `MULTI`/`EXEC` transactions. With Redix, we send these as a pipeline — there's no special transaction function: + +```elixir +iex> Redix.pipeline!(:redix, [ +...> ["MULTI"], +...> ["SET", "account:1:balance", "100"], +...> ["SET", "account:2:balance", "200"], +...> ["EXEC"] +...> ]) +["OK", "QUEUED", "QUEUED", ["OK", "OK"]] +``` + +All commands between `MULTI` and `EXEC` are queued and executed atomically — either they all succeed or none do. The real result comes back in the `EXEC` response (the last element), while the intermediate responses are just `"QUEUED"`. + +If we need to abort a transaction, we can use `DISCARD` instead of `EXEC`: + +```elixir +iex> Redix.pipeline!(:redix, [ +...> ["MULTI"], +...> ["SET", "key", "value"], +...> ["DISCARD"] +...> ]) +["OK", "QUEUED", "OK"] +``` + +## Best Practices + +There are a few things to keep in mind when using Redix: + ++ Always set appropriate TTLs on cached data to prevent memory bloat and ensure data freshness. ++ When executing multiple commands use pipelining for performance. ++ If we need atomicity across multiple commands, use Redis' transactions `MULTI`/`EXEC` for data consistency. + +## Conclusion + +Redix gives us a fast, resilient way to integrate Redis into our Elixir applications. Its direct use of Redis's native command structure means there's nothing extra to learn — if we know Redis, we know Redix. + +For caching with more built-in features like expiration policies and cache limiting, check out the [Cachex](/en/storage/cachex) lesson. If we need persistent storage without an external dependency, the [ETS](/en/storage/ets) and [Mnesia](/en/storage/mnesia) lessons cover what's built into the runtime. \ No newline at end of file