If you’re not familiar with warrant canaries, they’re brilliantly clever. Organizations publish signed statements saying they haven’t received secret government orders. Gag orders can prevent you from disclosing surveillance requests, but they can’t make you lie.

The beauty is in the negative space. If the canary stops singing, if the statement disappears or isn’t renewed, everyone knows something changed.

CanaryWatch became essential infrastructure. A central registry for monitoring these transparency signals. But like so many projects from that era, it eventually went quiet. The infrastructure aged. The tracking stopped. The canary itself went silent.

The need, however, never went away. In fact, the legal mechanisms compelling silence have only gotten more sophisticated. This is particularly true in jurisdictions that do not have the a 1st and/or 5th amendment protection that we have in the United States.

Today, I’m launching Canary Command, a modern platform for creating, signing, publishing, and monitoring (warrant) canaries. At its heart is the Extensible Canary Format (ECF), a new open standard that fixes the real problems with existing canary implementations.

The Problem With Today’s Warrant Canaries

Let me paint you a picture of the current state of warrant canaries.

Most are embarrassingly basic. A plain text file on a website, manually signed with GPG. No standardized format. No machine-verifiable structure. No automated verification. And absolutely no way for anyone to get notified when something changes.

Some organizations maintain them diligently. Others let them rot, forgotten files that become meaningless over time. There’s no ecosystem, no interoperability, no way to subscribe to updates.

This is infrastructure that should work even when you can’t speak. But the current implementations fail the moment someone gets busy or distracted.

Our tooling for privacy has not kept up.

I wanted to fix this and the result became the Extensible Canary Format (ECF) and the Canary Command platform.

ECF: Extensible Canary Format v1.0

At the heart of Canary Command is the Extensible Canary Format (ECF). It’s a complete specification designed from scratch to solve the real problems with existing canary implementations.

ECF documents are plain text that anyone can read and understand. But they also contain structured metadata that makes them machine-verifiable. Here’s what a simple ECF canary looks like:

--- BEGIN ECF STATEMENT ---

I, Jane Doe, hereby declare:
• I have NOT received any secret government orders
• I have NOT been compelled to provide access to user data
• I am making this statement of my own free will

This canary is valid until the expiration date specified in the metadata.

--- END ECF STATEMENT ---

--- BEGIN ECF METADATA (JSON) ---
{
    "version": "1.0-rc.1",
    "issued": {
        "by": ["Jane Doe"],
        "at": "2026-01-15T12:00:00Z"
    },
    "signers": [{
        "threshold": 1,
        "keys": [
            {
                "fingerprint": "A1B2 C3D4 E5F6 7890 1234 5678 9ABC DEF0 1234 5678",
                "weight": 1
            }
        ]
    }],
    "expires_at": "2026-02-15T12:00:00Z"
}
--- END ECF METADATA (JSON) ---

The metadata uses standard formats (JSON, YAML, or TOML), making it trivial for automated systems to parse, verify, and monitor. ECF supports weighted multi-party signing, so organizations can require multiple executives to attest before a canary goes public.

The Timing Problem

Here’s a subtle but critical issue with traditional canaries: cryptographic signatures prove who signed a document, but not when.

An adversary who compromises a signing key could theoretically create backdated canaries. Without proof of timing, you can’t trust that a canary wasn’t forged after the fact.

ECF solves this with Proof of Recency, an optional section containing publicly verifiable timestamped data:

--- BEGIN ECF PROOF OF RECENCY ---

Bitcoin block #925000 (2026-01-15):
Hash: 00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054

BBC News headline (2026-01-15 09:15 UTC):
"Tech leaders gather for annual summit in Davos"
https://www.bbc.com/news/technology-example

--- END ECF PROOF OF RECENCY ---

Including today’s Bitcoin block hash mathematically proves the document couldn’t have been created before that block was mined. It’s verifiable timestamping using public, immutable data.

On this platform, this proof data is a built-in feature. No manual work required. The platform supports Bitcoin blocks and news headlines from various RSS sources.

Multi-Person Verification

Corporate canaries often need multiple executives to sign off, ensuring no single person can be coerced into silence. ECF supports this with a weighted threshold system:

You can define multiple signers with different weights, set a threshold for validity, and even include “informational” signers who provide transparency but aren’t required.

For example, with keys weighted [3, 2, 1, 0] and a threshold of 3:

• The CEO (weight 3) can sign alone

• The CTO (weight 2) + CISO (weight 1) can sign together

• The PR director (weight 0) provides transparency but isn’t required

This creates flexible governance models where different stakeholders have different levels of authority. Canaries stay in draft until the threshold is met, keeping everything private until fully authorized.

All signing thresholds in the signer array need to be met.

Total Cryptographic Integrity

ECF’s security model rests on one fundamental principle: every signature covers the entire document.

When you sign an ECF canary, you’re cryptographically binding to everything, the statement, the metadata, the authorized signers list, the expiry date, even the proof of recency data. It’s all covered by the signature.

This creates ironclad security guarantees:

No backdating. The proof of recency is signed too. If it includes today’s Bitcoin block, the signature mathematically proves the document was created today.

No signer manipulation. The authorized signers list is embedded and signed. A bad actor can’t add themselves without breaking existing signatures.

No expiry tampering. The expiry date is signed content. To change validity periods, you need a new revision with all required signatures.

No silent modifications. Any change, even a single character, invalidates all signatures. The cryptographic binding is absolute.

Multi-party signing provides mathematical security. Even if one signer is compromised, they can’t forge signatures from others, modify the document undetected, or backdate the canary.

The integrity comes from decentralization and mathematics, not trust. It is highly recommended to also split signers across jurisdictions, which is further explained in the FAQ.

Built to Evolve

ECF is designed to grow with future needs. Organizations can add custom extension sections without breaking the core specification:

--- BEGIN ECF EXTENSION .v1.com.example.audit ---
[Custom audit data here]
--- END ECF EXTENSION .v1.com.example.audit ---

Extensions use inverted DNS naming to ensure global uniqueness. Parsers ignore extensions they don’t recognize, maintaining perfect forward compatibility.

Canary Command: Making It Real

The ECF specification is completely open, anyone can implement it. But I also built Canary Command, a full-featured platform that makes warrant canaries accessible to everyone.

From individual journalists to companies, Canary Command handles the complexity so you can focus on what matters: maintaining your transparency signals.

Zero-Knowledge, Zero Compromise

Your private keys never leave your GPG keyring. This wasn’t negotiable.

Canary Command’s desktop app (built with Tauri) invokes GPG commands directly on your machine. Private keys stay in GPG’s secure storage, never exported, never transmitted, never visible to our servers.

When you sign a canary:

• Isolated gpg --detach-sign command execution

• Private keys remain in your GPG keyring

• Only detached signatures and public keys go to servers

• No mechanism exists to access your private key

Even if our servers are compromised, your signing keys remain safe. It’s isolation between your keys and our infrastructure.

Two Ways to Deploy

Hosted Canaries: Perfect for getting started. We store your ECF document and provide the full signing interface. Use the desktop app’s native GPG integration or paste signatures manually. We handle revision history, notifications, and subscriber management.

External Canaries: For the control freaks. Host ECF documents on your own infrastructure, we fetch, verify, and monitor them. We become your independent watchdog. Signatures and public keys stay completely independent of platform accounts. Our domain verification ensures authenticity.

The Dead Man’s Switch (beta)

Here’s where it gets intense: poisoned keys.

A poisoned key is a pre-designated cryptographic dead man’s switch. If you’re coerced into signing while maintaining appearances, you sign with the poisoned key instead.

The canary immediately enters POISONED status, permanent, non-reversible, tamper-proof. All subscribers get instant alerts. The signal cannot be silenced or covered up.

Key properties:

• Non-retroactive: Only affects future signatures; past canaries stay valid

• Permanent: Once poisoned, it cannot be unpoisoned, edited, or archived

Pro tip: Prepare a dedicated duress keypair in advance. If compelled to sign under observation, using this key silently triggers the emergency signal while appearing to comply.

This is feature is currently in beta testing but available. It will be just one among more designed to be a failsafe / anti-coercion safety net.

The dual quorum rule

One particularly powerful feature is the dual-quorum rule, which provides robust protection against hostile takeovers. Given that the platform knows about the canary history, we can enforce that changes to a canary follow the previous revision signing threshold to allow modification and the new revisions signing threshold to publish that revision out of draft status.

This feature prevents hostile takeovers due to the fact that the signer list being a part of the canary itself within the structure metadata.

Personal Canaries: When Individuals Need Them Most

Warrant canaries aren’t just for organizations. Personal canaries may be even more critical for individuals facing elevated risk.

Journalists in hostile environments can maintain canaries affirming they haven’t been detained or had sources compromised. If a journalist goes silent, their canary’s expiration tells the story they cannot.

Activists and dissidents can signal continued safety. A regularly renewed canary proves ongoing autonomy. Its absence speaks volumes.

Whistleblowers can establish canaries before going public, creating cryptographic proof of their status. If targeted afterward, the canary becomes timeline evidence.

Travelers can set short-duration canaries before crossing borders with aggressive device policies. If compelled to unlock devices or install surveillance, the canary expires or gets poisoned, alerting contacts without requiring communication.

Security researchers can maintain canaries affirming they haven’t been approached to introduce backdoors.

The key insight: a personal canary isn’t just about legal orders. It’s proof-of-life for your digital autonomy. When you can no longer freely make the statement, the canary’s silence speaks for you.

The use-cases above and more are supported from the start. Your canary, your statement.

Teams and Individuals

Canary Command supports both personal workspaces (for individuals, activists, journalists) and shared workspaces (for organizations and teams). Multi-party signing enables secure collaboration with cryptographic accountability.

Personal workspaces are created automatically, sole access, full control. Shared workspaces use invite codes for team collaboration. You can import canaries between workspaces, preserving all history and subscriber relationships.

Draft management keeps revisions private until fully signed and approved.

Professional Templates

Not everyone knows how to write a legally sound warrant canary. We provide curated templates covering:

• Standard warrant canaries for organizations

• Transparency reports with request tracking

• Open source project canaries

• Privacy service templates for VPNs

• Personal safety templates for individuals

• Press freedom affirmations

• Australia compliant canaries (in theory)

Each includes customizable variables, optional clauses, and live preview as you build your statement.

Legal Disclaimer
The templates mentioned above are recommended for educational purpose. For more information, consult a specialized attorney in your jurisdiction.

Stay Informed, Stay Safe

Anyone can subscribe to public canaries and receive notifications when:

• New revisions are published

• Canaries approach expiration (configurable warnings)

• Status changes from active to warning to expired

• A canary gets poisoned (immediate red alert)

Delivery options include E-Mail, Telegram bot, and webhooks for integration with monitoring infrastructure. Never miss a critical update again.

Complete Audit Trail

Every change creates a new revision, an immutable ECF snapshot with full cryptographic integrity. Comprehensive audit capabilities include:

• Complete version history with diffs and syntax highlighting

• Side-by-side revision comparison

• Downloadable verification bundles for offline validation

• Full signer attribution and timestamps

• Immutable published revisions (permanent records)

• Flexible draft management

This creates a cryptographically verifiable chain of custody for every transparency statement.

Analytics Dashboard

Understand your canaries’ reach and health:

• Subscriber counts per canary

• Days since last update

• Status and time until expiry

• Revision history summaries

• Aggregate statistics across all canaries

Desktop App: Cryptography Made Simple

The web interface works great and is fully features. However, for a more convenient way of publishing canaries, there’s a native desktop application built with Tauri. It provides direct cryptographic operations with zero compromises.

The desktop app:

• Automatically detects and integrates with your GPG installation

• One-click ECF signing with isolated gpg --detach-sign execution

• Zero private key export, keys never leave GPG’s secure storage

• Real-time GPG status monitoring and debug logging

• Cross-platform compatibility (macOS, Windows, Linux)

GPG-optional by design: without GPG, paste signatures manually. With GPG, get seamless integration and one-click signing. Cryptographic operations become as simple as clicking a button.

Security: Your Keys, Your Responsibility

Canary Command includes comprehensive security documentation, but remember: the system is only as secure as your key management practices.

Key Storage Options

For maximum security, consider:

• Hardware security keys (YubiKey, Nitrokey), private keys never leave the device

• Air-gapped machines, dedicated signing computers never connected to networks

• Encrypted external media, keys on LUKS/VeraCrypt-encrypted drives

• Full disk encryption as minimum baseline

Geographic Distribution

Distribute key backups across multiple jurisdictions to resist single-point seizure. For advanced protection, use Shamir’s Secret Sharing, mathematically split keys into shares requiring multiple pieces for reconstruction.

Operational Security Basics

• Prepare dedicated duress keypairs for emergency signaling

• Plan for regular key rotation and lifecycle management

• Use multi-person signing to avoid single points of failure

• Regularly test backup restoration procedures

Legal Considerations

Documentation covers password vs. biometric protection, search warrant preparation, and secure key deletion. Always consult digital rights attorneys for jurisdiction-specific advice. This is educational, professional legal counsel is essential for high-risk scenarios.

Open Source, Open Standard

ECF is completely open source. I’m releasing the specification publicly because warrant canaries only work if the format is trustworthy, and trustworthiness requires transparency.

You don’t need Canary Command to use ECF. Parse it yourself. Build your own tools. The specification includes complete ABNF & EBNF grammars, detailed schemas, verification procedures, and security considerations.

Full specification: canarycommand.com/ecf-standard.md

Carrying the Torch

Canary Command isn’t affiliated with the EFF or CanaryWatch, but it exists because they showed us what was possible and necessary.

When CanaryWatch went quiet, it revealed a critical gap in digital privacy infrastructure. Warrant canaries haven’t become less important; the surveillance age has made them more essential than ever.

This is my contribution to that legacy: modern cryptographic infrastructure for transparency statements, built with the rigor the problem deserves.

What’s Coming Next

We’re actively building features to make Canary Command even more powerful. Here’s what’s planned:

Duress Passwords

Extending the poisoned key concept: secondary authentication credentials that trigger silent alarms. Use your duress password under observation, and the system appears normal while silently alerting subscribers. The digital equivalent of a silent alarm.

Mobile App

A native iOS app for high-risk individuals who need to manage canaries on the go:

• One-tap renewal when time is limited

• Biometric + PIN authentication (with disable option for high-security)

• Offline signing with sync when connectivity returns

• Secure enclave integration for hardware-backed key storage

• Travel mode with decoy interface for border crossings

• Push notifications for subscribed canary alerts

Critical for journalists, activists, and travelers who can’t always access desktop machines when they need to renew or signal danger.

The Canary Is Singing Again

Canary Command is live. Start yours today.

• Create your canary: canarycommand.com

• Read more on: canarycommand.com/faq

• Read the spec: Full ECF standard online

• Download the desktop app: Native GPG integration

• Subscribe to canaries: Monitor what matters to you

• Explore public examples: canarycommand.com

• Explore the platform canary

In a world of secret orders and gag provisions, cryptographic transparency isn’t optional, it’s essential.

The canary is singing again. Join the movement.

If you run a VPN, privacy company, open source project, news organization, or simply believe in the right to signal when you can no longer speak freely, this was built for you.

Alexander Weber
Chief Aviculturist of Canaries

January 2026

What makes qop different

At its core, qop manages versioned database changes with minimal bloat. Each migration lives in a timestamped folder containing two files: up.sql for applying the change and down.sql for rolling it back. This format is widely used—think Diesel migrations—but qop is designed to work with any stack, not just a single ORM or framework.

Highlights:

• Backend-agnostic: Works today with PostgreSQL and SQLite, with Cassandra support planned (among others).

• Simple migration structure: Timestamped directories with up.sql/down.sql ensure order and clarity.

• Lightweight CLI: One binary, no frameworks, no compile steps—just SQL.

• Built-in safety: Tracks applied migrations in a dedicated table, enforces linear history, and has recovery features for mismatched or missing files.

Where other tools try to be all things to all people, qop keeps its scope tight. You write the SQL, it makes sure it runs in the right order—and that you can undo it safely.

Installation

If you’ve got Rust installed, adding qop to your toolbox is a one-liner:

cargo install qop

If you’re working from the source repo:

cargo install --path .

That’s it. No extra dependencies beyond what your database already requires.

Initial Setup

Run this in your project directory:

qop init

This creates a qop.toml config file, where you define your database connection(s). For example, Postgres might look like this:

version = ">=0.1.0"

[subsystem.postgres]
connection = { static = "postgresql://postgres:password@localhost:5432/mydb" }
schema = "public"
table = "migrations"
timeout = 30

SQLite uses a file path instead of a schema:

[subsystem.sqlite]
connection = { static = "sqlite:///path/to/database.db" }
table = "migrations"
timeout = 30

Once configured, run:

qop subsystem postgres init

(or sqlite instead of postgres) to create the tracking table in your database.

Creating and Running Migrations

To create a new migration:

qop subsystem postgres new

You’ll get a folder like:

migrations/
└── id=1691875567123/
    ├── up.sql
    └── down.sql

Edit up.sql with your schema change and down.sql with the rollback. Then apply:

qop subsystem postgres up

This runs all pending migrations. Add --count <N> to limit how many are applied.

Rolling back is just as easy:

qop subsystem postgres down

By default, it undoes the most recent migration. Add --count to roll back multiple steps, or --remote to use the original down script saved in the DB (handy if local files have changed).

To see what’s pending:

qop subsystem postgres list

And if you want a dry-run preview of changes, use the experimental diff feature:

qop --experimental subsystem postgres diff

Advanced Features

qop isn’t bloated, but it does have a few tricks for real-world workflows:

• History sync: Creates local stubs for migrations found in the DB but missing locally.

• History fix: Reorders out-of-sequence migrations to maintain a strict linear history.

• Targeted apply/rollback: Apply or revert a single migration by ID without running others.

• Shell completions & manual: Generate completions for Bash, Zsh, Fish, or export the full help as Markdown.

How It Stacks Up

• Diesel CLI: Similar migration structure, but Diesel’s tied to its ORM and setup. qop is standalone, making it easier to drop into any project.

• Refinery: Offers library integration and more database support today. qop sticks to CLI-only and plain SQL for simplicity.

• SeaORM Migrations: Code-first migrations in Rust vs. qop’s SQL-first approach. If you prefer to write SQL directly, qop feels cleaner.

• Flyway/Liquibase: Broad database support and enterprise features—but heavier. qop feels more nimble for Rust developers who don’t need Java-based tooling.

My final thoughts

qop was built as an alternative migration tool because none of the existing ones felt right for me. It’s not trying to be the centerpiece of your database stack—it’s a rusty wrench you reach for when you need migrations handled cleanly. If you’re done wrestling with ORM migration quirks, or just want a lightweight, no-surprises solution, give qop a look.

It’s still young (0.4.x as of August 2025), but already capable. And with its open-source MIT license and active development, you can use it for whatever you’d like.

• crates.io

• GitHub repo

What does it do

In a nutshell, bobr executes multiple commands in the terminal at the same time (as sub process). That’s it.

Whether you want to just speed things up by executing them in parallel, or run multiple long running tasks in parallel as shown above. The example above shows two long running tasks (starting a HTTP server as well as the frontend).

bobr also forwards signals towards the processes started with it. That means that with a single “Ctrl+C” command (SIGINT), you can interrupt and abort all processes that run through bobr.

That’s it. No fancy magic or anything, just a straight forward simple quality of life tool that I built mainly because I was fed up with bash backend processes and bringing them back into the foreground etc.

Implementation

• Language: Rust
  My language of choice. Every day. All day. And night.

• Arguments: clap (clap_complete, clap_mangen, clap-markdown)
  It’s quite a big combination of crates here but it gives you full posix compliant argument parsing, a great interface to work with and manpages + markdown manuals

• Async runtime: tokio
  tokio is the #1 async runtime library

• Signal handlers: signal-hook
  Makes it super simple to list for signals like SIGINT ot SIGTERM

• Cross-thread message channels: flume
  Simple messages across thread boundaries with a mpsc architecture

• Error handling: anyhow + thiserror
  The easiest way to handle errors properly, without need for complex custom error types

• Terminal manipulation: crossterm
  Great for manipulation terminal output incl. coloring, clearing screen, moving cursor and much more

Being able to identify (and measure) quality of any work product is always positive. It helps in delivering a product that suits the customers, lets you articulate the reason for software changes and is, in some cases, required to meet specific standards either by policy or law.

Today, we’re looking into what this term actually means.

Dissecting Software Quality

A broad term like software quality needs to be dissected into smaller, interpretable pieces in order to work with it effectively. For this, I would like to introduce one way of doing this using a well defined ISO standard.

Meet ISO 25010:

Called “Software Product Quality”, this standard aims to dissecting software quality into many smaller, categorized sub-concerns.

Ranging from functional suitability through usability to portability, this standard covers many aspects that are relevant for any software.

As an introduction, here a quick overview about the given categories:

• Functional Suitability
  aka does the program do what it is supposed to do

• Performance efficiency
  aka how does my program make use of the available resources

• Compatibility
  aka how nice is my program working together with other programs

• Usability
  aka how frustrated does my program make its users

• Reliability
  aka does my program do its thing when I want it to do it, and what happens if it doesnt

• Security
  aka security

• Maintainability
  aka how frustrated does my program make its developers

• Portability
  aka how tight is my program coupled to what it’s running on

You will notice that Security does not really have a short form of definition since I think that all attempts to reduce it in size and scope can have severe implications and right now, let’s just roll with this term.

Are all equally important?

As so often, the answer is “it depends”. It depends on the environment. But luckily, we can induce an intrinsic ranking of some of the quality disciplines.

Functional Correctness

If the software does not do what it should do, all other aspects are irrelevant. Let’s say you’re writing a software with textbook code quality. Right abstractions, meaningful variables names, optimal time complexity etc. Only problem is that instead of a microservice for billing you built a CLI interface for Reddit.
It is indeed an extreme example but that should make the idea behind this very clear.

Usability

No customer, no money, no developer, no product, no company, no yacht. Customers are the fundamental concerns of every business (or at least should be) and it is everyones responsibility to champion their needs. In that sense, if your software is not usable by a customer, it wont be used and therefore it is waste.

The rest

I’ll cap it here as most other aspects can be mitigated. Some over-the-top examples to show what I mean:

• Software is running out of memory due to wasteful algorithms?
  An economical solution could be to just add more memory (for now).

• Software is not installable on a specific system?
  Choose a different system it can be installed on.

• Code quality is horrendous and unreadable?
  You’ll have a high mean time to change but this is solvable with enough money and time.

Obviously, all the things above are clearly extremes but this should transport the general idea behind it.

In any case, what you want to achieve is a balanced performance through any of the categories while keeping in mind that the optimal engineering solution is not always the most economical solution.

Security

I have not forgotten about this as it deserves it’s own little section. As for all of the aspects above, you want to achieve a base-line of quality in this section, too.

With regards to security, the implications can be severe though. Therefore, as economical as we want to be, it is our responsibility as engineers to make sure that minimum security requirements are met and monitored. This is an aspect that is sometimes valued as less important, especially by non-technical leadership, but it is a critical component of any software.

With that being said, I will encourage any engineers to stand their grounds against malpractice regarding security aspects. As a technical expert, no one is going to champion it if you don’t.

• Do not “worry about authentication later”
  (actual statement I had to argue against more than only once)

• Authorization is not “just an attribute”

• Audit Logging is no automatical by-product
  Consider properties like append-only and decentralization / copies. Who can access it? What is logged? …

• MFA should not just be an “extra”

• And please … start signing your Git commits.

Closing thoughts

Software quality is often a quite blurry topic and this article does not even come close to defining it properly. It mainly is just a collection of my thoughts on the topic looking back at a 13 year career.

Code quality is not equal to software quality.

Customer obsession. It is one of the Leadership Principles at Amazon for a good reason. No customers, no company, no income. It is that simple.

I think in the end it is always important to look at software as a tool. Make your customers want to use your tool over the other ones. And when building the tool, we as developers sometimes have to stand our ground to adhere to minimum standards, making the tool safe to use, even if it costs more to produce.

You probably don’t need to implement the Levenshtein distance between two strings every day. Given that most of the engineering work is CRUD apps that follow a pre-established pattern anyway (especially in more junior ranks and bigger companies), the effectiveness of this interview practice might be debatable. Nevertheless, companies ask these questions for good reasons and it has shown that the process, as cumbersome as it may sometimes be, is working(*). Therefore, we have to face the reality of having to deal with these types of questions. I myself have been on both sides of the screen and want to share my thoughts about going back to the grind.

TLDR;

There is no magic formula or secret trick. In order to become better, nothing beats hard work. No one is going to do it for you.

After some time you will develop a sense / feeling for which classes of Algorithms are viable for which sets of problems. Trust your gut, it will mostly be right.

Timebox solutions

Solving questions can be pretty hard at first, especially when not having been exposed to a variety of different problems and solution approaches.

When starting out, timebox yourself to 15-20 minutes for coming up with a plan / solution. Many problems have multiple solutions. When starting out, come up with a plan and verify that the solution is viable.

If it is, implement it and get it working. Now delete it and implement it again. Repeat this until you understand the approach.

Keep notes

I use Obsidian for my note taking. In there I have a folder called LeetCode. I keep notes for all questions I solve and label them with the difficulty and a “redo” flag.

For all problems, I store the problem statement, solution code and a description in which I explain the solution to myself. Don’t be lazy on that last part, this is the most important one.

Forcing yourself to explain the solution triggers the rubber duck effect and will help you understand and remember it better.

Repetition

On most days, I randomly pick 2-3 problems and just simply redo them. Practice is king and will go long ways.

With enough practice and learning / repeating the optimal solutions to problems, you have no choice but to better understand and remember them.

Algorithms will repeat

The more coding questions you solve, the more often you will remember something similar you’ve done somewhere else.

Most questions are solved using common techniques. This includes dfs and bfs for graph problems (most common asked interview questions), matrix (and array) traversal and manipulation, sorting and dynamic programming (dp).

For example, solve and compare LC-1 and LC-1010. Do you find similarities?

Many times, with practice, you can early identify what solution will be viable just by the feeling you develop for these kinds of problems. Trust your gut instinct and go with it.

I would confidently say that you can solve 80% - 90% of problems with common techniques once you identify what class this problem falls into.

Dynamic Programming sucks

Dynamic Programming problems are notorious for being just a pain. A lot of companies stopped asking these questions. Although I’d still encourage you to learn them, I found that the time required to do so is mostly better invested solving other problems. This, of course, only holds as long as you have relevant other questions on top of the pile.

If not, well… No one’s going to solve the problems for you.

Buy Premium

I know, it costs money. But trust me, just do it. Don’t doubt it, it’s worth it after all.

Alex out

The application we’re building is going to send commands from a container to the host, having the host execute them. Obviously, this is a potentially critical operation that escapes the container isolation but it’s a great example for how to send any kind of messages towards the host (or back).

Stack

• NixOS
  is a declarative Linux distribution that let’s us achieve reproducibility for our environment. It will be used as the host operating system.
  We ignore windows, it does not exist.

• podman / podman-compose
  podman is a runtime for OCI containers much like docker. It supports rootless execution and otherwise is (almost) fully compliant to docker.

• Rust
  programming language of choice (just a small excerpt)

Host setup

The following configuration is an excerpt for a NixOS host configuration that can be used to host our containers.

{
  # ...

  # enable virtualisation and podman as backend
  virtualisation = {
    containers = { enable = true; };
    oci-containers = {
      backend = "podman";
    };
    podman = {
      enable = true;
      autoPrune = { enable = true; };
      dockerCompat = false;
      defaultNetwork = {
        settings = {
          dns_enabled = true;
        };
      };
    };
  };

  # make sure podman and podman-compose are installed
  environment = {
    # ...
    systemPackages = with pkgs; [
      # ...
      podman
      podman-compose
    ];

  # systemd services
  systemd = {
    services = {

      # startup service
      # ensures that there exists a named pipe we can use
      startup = {
        enable = true;
        wantedBy = [ "multi-user.target" ];
        
        serviceConfig = {
          Type = "oneshot";
        };

        script = ''
          mkdir /etc/example || true
          mkfifo /etc/example device/cmds || true
          chown -R example /etc/example
        '';
      };

      # listens on the named pipe, executing all commands sent through it using the eval command
      host_commands = {
        enable = true;
        wantedBy = [ "multi-user.target" "startup.target" ];

        serviceConfig = {
          Type = "simple";
        };

        script = ''
          while true; do eval "$(cat /etc/example/cmds)"; done
        '';
      };
    };
  };
}

• systemd.services.startup
  This is the service that bootstraps the named pipe at “/etc/example/cmds“ if it does not exist.
  This service spawns at system startup (“multi-user.target”).

• systemd.services.host_commands
  This is the service that continuously reads from the named pipe at “/etc/example/cmds“ and executes the commands sent through it using the eval command.
  This service spawns at system startup (“multi-user.target”).
  Not that this service depends on the startup systemd service.

Agent software

Rust code

The following code will write the command “printf 'ping' >> /etc/example/container.log” to the file “/app/cmd” every two seconds.
Not that in order to be able to write to the pipe, we only open the file using with write access as we won’t read from it and just use it as a sink.

let mut fifo = std::fs::OpenOptions::new().write(true).open("/app/cmds")?;

loop {
  fifo.write_all(b"printf 'ping\n' >> /etc/example/container.log")?;
  fifo.flush()?;
  
  tokio::time::sleep(Duration::from_secs(60)).await;
}

compose.yaml

In this yaml file, we build and run the agent software. The important part here is that we mount the file “/etc/example/cmds“ into the container at path “/app/cmds“. This file is a named pipe in the host fs.

services:
  agent:
    build:
      context: ../
      dockerfile: docker/Dockerfile
    container_name: agent
    restart: always
    volumes:
      - /etc/example/cmds:/app/cmds # this is the named pipe mount
    environment:
      DOTENV: "0"
      RUST_BACKTRACE: "1"

Result

Running the container on the host system will use the file at “/etc/example/cmds” in order to exchange from the container to the host system. In our use-case, this data is shell commands that are then read and executed by the “host_commands” systemd service. In the given example, this results in the string “ping\n” being appended to the file at “/etc/example/container.log” every two seconds.

Disclaimer: Again, this is a pretty niche thing to do with containers and has obvious security implications but sometimes, use-cases like this can present themselves so why not. Don’t blame me if someone runs “rm” through that channel though.

Assumptions

• Existing Hetzner account with access to Robot servers

• Existing Robot server

Step 1: Rescue mode & Talos install

First, go to the Hetzner Robot console and go to the “Rescue” tab. Select and SSH key from your known keys and press “Activate Rescue System”.

The server is now going to restart into the rescue OS, a specialized type of operating system that is used to rescue machines that are broken in some way. We are going to use this to override the filesystem of the machine with Talos Linux.

After a reboot, connect to the machine via SSH, using the “root” user. Now perform the following steps — you might need to update drive names. In this example, I had “nvme0n1” and “nvme1n1”.

# Disable raid
mdadm --stop /dev/md[0-4]

# Wipe disks
sfdisk --delete /dev/nvme[0-1]n1
wipefs -a -f /dev/nvme[0-1]n1

# Download raw talos fs
wget https://github.com/siderolabs/talos/releases/download/v1.7.0/metal-amd64.raw.xz -O /tmp/talos.xz
# Replace system with talos
xz -d -c /tmp/talos.xz | dd of=/dev/nvme0n1

sync
reboot

This script will now reboot the machine with Talos Linux as operating system.

Step 2: Talos bootstrapping

talosctl gen config my-cluster-name https://[INSERT IP HERE]:6443

# Edit the '/dev/sda' drive in the control-plane.yaml to '/dev/nvme0n1'
vi ./control-plane.yaml

talosctl apply-config -f controlplane.yaml -n [INSERT IP HERE] -e [INSERT IP HERE] --insecure
export TALOSCONFIG=./talosconfig
talosctl config endpoint [INSERT IP HERE]
talosctl config node [INSERT IP HERE]
talosctl dashboard
talosctl bootstrap --talosconfig=./talosconfig
talosctl kubeconfig ./kubeconfig

export KUBECONFIG=./kubeconfig

# Optional: Upgrade for ISCSI support (needed for Longhorn)
# --preserve is needed for single-node clusters
talosctl upgrade --image factory.talos.dev/installer/613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245:v1.7.1 -m powercycle --preserve

# Allow scheduling on control-plane node for single node clusters:
kubectl taint nodes --all node-role.kubernetes.io/control-plane-

Disclaimer

This tutorial is not meant to create production clusters. It is a bare minimal example of how to get started with an affordable, single-node Kubernetes cluster on a dedicated Hetzner machine. For production use-cases you will need HA for control-plane and workers, verifying checksums when downloading files, additional firewall configurations etc. etc.

Happy hacking.

As a common problem in systems, especially ones that are built according to service oriented design principles, this problem has been solved many times and in just as many different ways.

This is a post about a reference implementation I came up with and successfully used in many projects. The shown solution has been scaled to millions of callers, been extended to individual needs and proved stand the trials of entire product lifetime cycles.

General Concept

API keys are created server-side and delivered to the API caller. The key is then transformed (see below) and stored in a secure data store, only accessible to the authority and verifier.

The stored data MUST allow referencing but MUST NOT allow reconstruction of the key itself. The secret part is and will only be known by the caller of the API and MUST NOT be known to the API authority after it has been transferred to the customer. It’s basically the same reason why a system should not know the plain password for a user account.

API requests MUST carry the key with each request. In a HTTP setting, you would usually carry the API Key in the request header field under Authorization x-api-key or some other field name. Consider that if used under the Authorization header, you might want to prefix the key with some identifier to distinguish it from JWTs that are carried in the same field. This is common practice for APIs supporting multiple authentication paradigms.

Key creation

1. API authentication authority creates API key pair

   1. It creates a unique key ID

   2. It creates a secret part

2. Storage

   1. The secret part is transformed through the argon2 key derivation function. The parameters of this function should be appropriate.

   2. The key is stored in the data store. Fields include the Key ID (plain text), argon2 secret hash and optional information like the tenant, billing account, expiry time etc.

3. Delivery

   1. The key is concatenated to a single string using a delimiter character. This caracter is not part of either the ID or Secret generation alphabet and must be unique inside the concatenated key.

   2. The concatenated key is transmitted to the caller.

   3. The caller verifies the API key, setting it from pending to active.

API calls

1. The caller initiates a call to the API with the key contained inside the call. For HTTP APIs, this would be some header field.

2. The callee takes the key from the request and splits it into ID + Secret.

3. The callee performs a lookup of the Key ID in the data store and receives the information stored alongside it.

4. The callee verifies the argon2 hash from the data store with the secret from the API request.

5. IF NOT successful, the callee denies the call as not authenticated / unauthorized.

6. IF successful, the callee scrubs the API key information from the request and forwards it to the handler that is responsible for executing the request.

Key rotation (optional)

Key rotation is a critical security feature and SHOULD be implemented by any API that supports key authentication.

1. Caller initiates key rotation with old key (A) via API call

2. see Key creation

   1. Mark current key (B) as pending (can only be used finish rotation)

   2. Also, mark B as successor to A

3. Call rotation completion endpoint with new key

   1. Marks A as deprecated

   2. Marks B as active

4. Perform other API calls from here exclusively with B.

Structure

The API Key structure is as follows:

<API_KEY_ID><DELIMITER><API_KEY_SECRET>

1. API_KEY_ID
   A unique identifier of the API key. This information is known to the API authority as well as the consumer. It can be used to perform service, debugging, tracing and other operations. It is not security critical.

2. DELIMITER
   Some delimiter that’s not a part of the character sets of the KEY_ID and KEY_SECRET generation algorithms.

3. API_KEY_SECRET
   The secret part of the API key. This is a security critical information and MUST NOT be logged, displayed, stored or forwarded in any way.

The overall structure here is quite similar to HTTP basic auth, concatenating username and password in the request.

Considerations

Middlewares and authority

In any API implementation, it is probably a good idea to split the responsibility for the auth subsystem into a separate unit. This could look different depending on the setup. Two options are an entirely separate service that acts as authorizer on API routes (let’s say for HTTP APIs) or a combination of middleware (verification) and authority (sub-router routes with handlers) inside of the service for a more monolithical approach to SOA. Both have advantages and disadvantages.

What these have in common is that between the API Gateway and the actual request handler containing the business logic, the API key is verified and the request is approved. In most cases, you want to have some information from the API key available to the request handlers. This can be done by enhancing the downstream request with the information needed by the request handlers. For example, AWS API Gateway Authorizers allow modification of the HTTP request towards downstream handlers.

In any case, the original API Key MUST be scrubbed from the request as it contains the original plain-text secret. This information MUST NOT be accessible outside the secure enclave of the auth system.

Storage

The API Key MUST be stored in a secure data storage. The secret is and will only be known by the caller. It generally is to be considered the same criticality as user passwords.

In order to achieve the requirement of allowing lookup but disallowing reconstruction, we store the Key ID in plain text as primary key in the data store. The secret part goes into some trap-door algorithm that allows us verification but disallows reconstruction of the original value.

For storage, consider the following alternatives:

1. Dynmao like

   1. DynamoDB

   2. ScyllaDB

2. Redis like

   1. Redis

   2. KeyDB

In any case, since the information must be read for each request, you want to consider a fast alternative that allows quick reads when having the Primary Key at hand.

DynamoDB for example gives a single-digit millisecond guarantee on a lookup with high entropy on the partition key (in our case that’s the Key ID).

An API key struct could look as follows (pseudo data structure):

#[derive(Debug, serde::Serialize, serde::Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum ApiKeySecret {
  Argon2(String),
}
impl ApiKeySecret {
  pub fn verify(&self, secret: &str) -> Result<()> {
    todo!("implement");
  }
}

#[derive(Debug, serde::Serialize, serde::Deserialize)]
#[serde(rename_all = "snake_case")]
pub struct APIKey {
  pub id: String, // primary key
  pub secret_hash: ApiKeySecret, // key secret
  
  // use-case specific (optional) fields
  pub tenant: String,
  pub expires_at: Option<DateTime<Utc>>,
  // ...
}

Security concerns

Incoming plain-text API keys MUST NOT be logged in their raw form as this would expose the secret in the logs. The secret MUST be omitted / replaced when doing so. Pro tip: In Rust, you can implement the Debug (…) trait to customize how a key is logged, avoiding accidental leaks here.

API key secrets, like passwords, SHOULD be rotated on a schedule. Contrary to passwords, regular schedules do not lead to predictable patterns in the key since the secret is generated server side. This SHOULD be done with a high entropy in the secret generation algorithm, avoiding predictable patterns (like auto-incrementing numbers…).

Since the callee authority itself does not know the plain secrets of the API key, we already mitigate a whole set of concerns about unauthorized access. Nevertheless, altered information in the API key tables can have other implications regarding billing, metering and other business cases.

Additional security can be achieved by alerting on key rotations, implementing intrusion detection mechanisms that alert on unusual behavior and significantly restricting developer access to the production system, especially considering the secure enclave containing the key information.

Argon2 key derivation function

Argon2 is a key derivation function that has shown to be resistant to cracking attempts.

A key aspect of the password hashes is that the parameters used are stored inside the hash. This is a great feature because it allows changing the algorithm in the code while still supporting older secrets being validated.

API keys vs. JWT

JWT stands for JSON Web Token. A JWT is a very common authentication mechanism between a frontend (interactive user interface) and a backend service (like a RESTful HTTP API). Although JWTs can be used for other situations, its most common occurrence is in this setting.

The great feature of JWTs is that it enables us to store information inside the token and verify the integrity and validity of this information server side. This avoids database lookups for each request. The token is created by minting it server side, enclosing it on every HTTP call and then verifying the checksum of the token during the call on the server side. This can be done in many ways, with signed cookies being the most prominent one for ordinary frontends.

Given that the information is stored in the token, there are a couple of downsides to this approach. For one, the information in the token generally not encrypted (so it’s readable to the user). If the token stores information about the internal system, like authorization claims, this can expose information to the public that could otherwise stay hidden. Although you should not perform security by obscurity, this does technically serve as an additional attack vector. Also, once minted, JWTs can not easily be revoked, making long lasting tokens (like refresh tokens) more critical due to their blast radius. Once a token has leaked, it can be used to mint new tokens with the previous ones still being valid. It’s a downward spiral from here.

Generally speaking, JWT is better suited for user authentication to a system while API keys are better suited for system to system authentication (exceptions apply).