Skip to content

Add setTargets script for bulk assessment targets - #11

Open
JaredEzz wants to merge 4 commits into
mainfrom
CU-868kervpp_Python-script-for-mass-setting-of-targets
Open

Add setTargets script for bulk assessment targets#11
JaredEzz wants to merge 4 commits into
mainfrom
CU-868kervpp_Python-script-for-mass-setting-of-targets

Conversation

@JaredEzz

@JaredEzz JaredEzz commented Aug 4, 2026

Copy link
Copy Markdown

https://app.clickup.com/t/868kervpp

new setTargets/ script for setting assessment targets in bulk, for customers who don't want to click through every control one at a time.

  • --template [CSV] writes every aspect the key can reach with its current target. won't clobber an existing file without --overwrite, so a half-filled template survives a re-run
  • --apply groups the filled-in rows by assessment and sends one PUT /v1/businessUnits/{businessUnitId}/assessments/{assessmentId}/targets per assessment
  • fails closed per assessment. the endpoint saves a request as a unit, so any rejected row (duplicate aspectId, unknown aspectId, unparseable target, bad weight) drops that whole assessment instead of writing the rows around it
  • reports who already holds targets on the assessment and prompts, because an aggregate business unit averages targets across managers instead of replacing them
  • re-reads /scores after each write and requires target and weight to come from the same contributor before calling a value present
  • key comes from OSTRICH_API_KEY or a hidden prompt. unattended --apply without --yes is refused up front rather than hitting an unanswerable prompt mid-run
  • --dryRun validates and prints without writing. exit status is non-zero for any rejected row, skipped assessment, failure, or value that didn't read back
  • setTargets/README.md covers key creation, the Manager role requirement, csv format, the Excel save-as gotcha, and aggregate vs override

exercised against a stubbed api across 11 cases, then run live against a real dev tenant, which caught two real bugs the mock hid (see the collapsible below). both are fixed and reverified live. one live-only finding is a backend defect, not a bug in this script, also below.

AC

  • generate a key with Manager on a test bu, run --template, confirm one row per aspect and that rollup assessments are absent
  • re-run --template, confirm it refuses; then --overwrite, confirm it replaces
  • fill a few rows, --apply --dryRun, confirm nothing is written
  • drop --dryRun, confirm the targets show in the app and every value reads back
  • put the same aspectId twice in one assessment, confirm that assessment writes nothing (not just the second row)
  • put a bogus aspectId in the csv, confirm that assessment is skipped and no request is sent
  • run with a key holding only practitioner roles, confirm the 403 comes back and the message names the missing role
  • try one against a closed assessment, confirm the 400 is reported and the run continues
  • OSTRICH_API_KEY=... python setTargets.py --apply f.csv with no tty and no --yes, confirm it refuses before writing anything
  • point --apply at set targets example.csv with the ids swapped for real ones, confirm the minimal 5-column form works
two bugs the mock hid, found and fixed by testing live against dev

the mock in the earlier review faked responses. testing against the real dev api (apid.ostrichcybersecurity.com) with a real Manager-scoped key surfaced two real bugs the mock's wrong assumptions had hidden, both fixed and reverified against the real api:

  • save_targets() crashed with KeyError: 'response' on every real, successful save. the targets endpoint only ever returns {"message": ...}, no response envelope, unlike every read endpoint. confirmed live: the write always landed correctly server-side, only the client crashed reading its own success. fixed by making the envelope check opt-out per call instead of universal.
  • an unattended run that hit the shared-targets prompt with no answerable stdin caught EOFError and returned False, so it printed "treating as no" and exited 0 having saved nothing, indistinguishable from a deliberate skip. now raises and exits non-zero. found this because sys.stdin.isatty() reports True under < /dev/null redirection on this platform (windows git bash + native python), so the early refusal never fired and the run reached the actual prompt. --dryRun no longer reaches the prompt at all now either, since a dry run should never block on an answer it doesn't need.

one open, known limitation from the same pass: if one assessment already saved before a later one in the same run hits this exact eof case, the run exits without printing the usual saved-count summary. the values already saved are correct and unaffected, only the summary line is missing. not fixed here, flagged for a follow-up.

found live, not a bug in this script: businessUnits list 500s for a nested-bu-scoped key

GET /v1/businessUnits/ (the first call --template makes) 500'd against the real dev api for a key scoped to a business unit that isn't top-level in firestore, reproduced with a raw curl outside the script entirely. every other endpoint (assessments, content, scores, the targets put itself) resolved the same nested bu correctly via its path parameter. a key scoped to a subsidiary or department rather than the company root is a completely normal way to scope a customer key, so this blocks --template for exactly that case.

this is a backend defect, not something in this PR's diff, and out of scope to fix here. worth its own ticket against emu.

notes: the readme describes roles, on purpose

the readme states the requirement as the Manager role on the business unit and does not go below that. that is what the deployed api checks and what its 403 names, and it stays accurate as role handling changes, so the readme should not need rewriting later.

the closed-assessment row quotes Cannot save scores or targets on a closed assessment for the same reason. that is the message the shared scores-and-targets handler returns, ahead of the targets-only mapping further down.

why the aspect list comes from two calls

/scores returns one entry per function, category, question and aspect. the nodes map in /content holds every function, category and question, so removing those keys and the summary row leaves the aspects, which are the only ids that accept a target.

matching a pattern against the ids would also work for the built-in frameworks, where an aspect id like GV.OC-1-PROCESS has one more hyphen segment than the question GV.OC-1. custom content types don't have to follow that shape, so the script derives the set instead.

aggregate vs override, and what the read-back check can and cannot prove
strategy how a target is stored
aggregate one target per manager per control, and the displayed value is the average
override one target per control for the whole assessment

targets are attributed to the account that created the key, not to the key itself. on an aggregate business unit that means a bulk load replaces that user's own targets and leaves every other manager's in the average, so the same csv gives a different result depending on who runs it. the script reads the existing contributors out of targetLabels and the per-sub target values, names them, and asks before writing.

the read-back check afterwards confirms a requested value is present on the assessment, not that it is attributed to the caller. the public api doesn't expose the calling account's own subId, so per-user verification isn't available; requiring target and weight to match one contributor closes the worst version of it, where one manager's target paired with another's weight read as a match. the readme says this in as many words rather than claiming more than it checks.

switching the strategy afterwards is not neutral either. aggregate to override carries the averaged values over, but only on assessments that have never held override targets before, otherwise the older override values resurface. override to aggregate carries nothing over.

notes
  • ostrichApi/ is vendored per folder, matching exportAssessmentStatus/ and createScenarioFamilies/. no shared package.
  • template files are written utf-8 with a BOM so excel round-trips them as utf-8; reads use utf-8-sig either way, and a non-utf-8 file gets a save-as instruction rather than a stack trace.
  • decimal targets like 75.0 are rejected rather than coerced. guessing at a malformed value is worse than naming the row.
  • rows with both target and weight blank are skipped, so a partly filled template is a valid input.
  • every request has a 180s timeout and --baseUrl has to be https, since the key and the token both cross it.
  • requests is pinned at 2.34.2.
  • root README.md now indexes the three script folders, since customers land there from a shared link.
  • adversarially reviewed cross-vendor before this went up. the fail-closed batching, the prompt handling and the read-back semantics above all came out of that pass.

JaredEzz and others added 3 commits August 4, 2026 15:23
setTargets reads every aspect an API key can reach into a CSV with its
current target, and writes filled-in targets and weights back through
PUT /v1/businessUnits/{id}/assessments/{id}/targets.

Before saving it checks the aspect ids against the assessment, rejects
duplicates, and reports who already holds targets there, since an
aggregate business unit averages targets across managers rather than
replacing them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The deployed API gates the public save-targets route on the Manager role and
its 403 names that role. Quoting a permission string described a state that
is not live, and the closed-assessment rejection comes back worded for both
scores and targets.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The targets endpoint saves a request as a unit, so a batch that holds a
rejected row must not be sent at all. A duplicate or unparseable row now
drops its whole assessment rather than quietly saving the rows around it.

Other ways a run could mislead or strand someone:

- The confirmation prompt reads the same stream as the API key, so a
  piped key made it unanswerable partway through a run. The key comes
  from OSTRICH_API_KEY or a hidden prompt, and an unattended run without
  --yes is refused before the first write.
- --template refused nothing when overwriting, so a filled-in template
  could be lost. It now needs --overwrite, and takes a path.
- A template written from a partial read exited zero while claiming to
  hold every aspect. Unreadable assessments are listed and exit non-zero,
  and business units the key holds nothing on are counted.
- The read-back check matched a target from one contributor against a
  weight from another. Both now have to come from the same one, and the
  wording no longer claims the value is attributed to the caller.
- Requests had no timeout, --baseUrl accepted plaintext, and the key was
  echoed while being typed.
- A header carrying neither target nor weight reported nothing to do.
- Template files are written with a BOM so Excel round-trips them as
  UTF-8, and a non-UTF-8 file gets an instruction rather than a stack
  trace.

flatten_business_units builds display paths on the way down instead of
storing parents on the payload, which keeps the API response free of the
reference cycles that made it untestable.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@JaredEzz
JaredEzz marked this pull request as ready for review August 4, 2026 21:54
Both survived earlier review because the mock assumed a response shape
the real API doesn't return.

- save_targets() crashed with KeyError: 'response' on every real,
  successful save. The endpoint only ever returns {"message": ...}, no
  "response" envelope; _request() now takes has_response_body and
  save_targets() opts out of the envelope it never gets. Confirmed
  live: the write always landed server-side, only the client crashed
  on the success response.

- confirm_shared_targets() caught EOFError and returned False, so an
  unattended run that couldn't answer the prompt printed "treating as
  no" and exited 0, having saved nothing, indistinguishable from a
  deliberate skip. It now raises SetupError so main() reports exit 2
  instead. --dryRun no longer reaches the prompt at all: it states who
  already holds targets without asking, since a dry run should never
  block on an answer it doesn't need.

Verified against the real dev API, not the mock: the exact save that
used to crash now completes and reads back correctly, a
duplicate-aspect batch still writes nothing, and the EOF path fails
loudly instead of silently.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@JaredEzz
JaredEzz force-pushed the CU-868kervpp_Python-script-for-mass-setting-of-targets branch from 4ad5780 to 5dbceda Compare August 5, 2026 21:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant