Skip to content

fix(comparisons): make sandbox and session-policy rows binary #6524

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
waleedlatif1 wants to merge 1 commit into from
Closed

fix(comparisons): make sandbox and session-policy rows binary #6524

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions apps/sim/lib/compare/data/competitors/claude-cowork.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -647,10 +647,10 @@ export const claudeCoworkProfile: CompetitorProfile = {
},
codeSandboxRuntime: {
value:
'Partial: shell commands and any code Claude writes run in an isolated sandbox (a dedicated Linux VM for local sessions, a per-session temporary sandbox on Anthropic-managed infrastructure for cloud sessions), and within a session Claude can itself install dependencies at run time from approved package managers (npm/registry.npmjs.org, PyPI/files.pythonhosted.org, GitHub, crates.io, Yarn). What an administrator configures is the org-wide network-egress policy rather than the environment: Team and Enterprise organization owners choose, in Organization settings > Capabilities, between network access off (Claude "operates with pre-installed packages only"), package managers only, package managers plus an admin-specified domain allowlist, or all domains except Anthropic\'s legal blocklist.',
'Yes: at the package layer only, and driven by the agent rather than a declared environment. Shell commands and any code Claude writes run in an isolated sandbox (a dedicated Linux VM for local sessions, a per-session temporary sandbox on Anthropic-managed infrastructure for cloud sessions), and within a session Claude can itself install dependencies at run time from approved package managers (npm/registry.npmjs.org, PyPI/files.pythonhosted.org, GitHub, crates.io, Yarn). What an administrator configures is the org-wide network-egress policy rather than the environment: Team and Enterprise organization owners choose, in Organization settings > Capabilities, between network access off (Claude "operates with pre-installed packages only"), package managers only, package managers plus an admin-specified domain allowlist, or all domains except Anthropic\'s legal blocklist.',
detail:
'The configuration surface is an org-wide network-egress policy plus run-time installs the agent performs itself, not a declared environment: there is no manifest of packages, no OS/system-package list, no preinstalled-CLI selection, and no custom base image or Dockerfile for the sandbox, and Anthropic does not publish the preinstalled package set for Cowork sandboxes. Network settings apply when a session is created, so changing them mid-conversation requires starting a new one.',
shortValue: 'Partial: agent installs packages at run time; environment not declarable',
shortValue: 'Yes: agent installs packages at run time; environment not declarable',
confidence: 'verified',
sources: [
{
Expand Down Expand Up @@ -928,10 +928,10 @@ export const claudeCoworkProfile: CompetitorProfile = {
},
sessionPolicy: {
value:
'Partial: the setting is available only to "Admins and Owners of Enterprise plans and Console Admins," who can enable a shortened session length of 1, 7, 14, or 28 days (Console Admins get 1, 3, or 7 days), after which "users will need to sign in again after the specified period, even if they\'ve been actively using Claude." This is an absolute lifetime cap from sign-in, not an inactivity timeout, and no idle timeout is documented.',
'Yes: on Enterprise plans only. The setting is available only to "Admins and Owners of Enterprise plans and Console Admins," who can enable a shortened session length of 1, 7, 14, or 28 days (Console Admins get 1, 3, or 7 days), after which "users will need to sign in again after the specified period, even if they\'ve been actively using Claude." This is an absolute lifetime cap from sign-in, not an inactivity timeout, and no idle timeout is documented.',
detail:
"Because the control is Enterprise/Console-gated, administrators on other plans cannot set a session lifetime at all; with the setting disabled, sessions return to the default behavior of remaining active as long as the user stays active, with no documented idle timeout. Because one session spans all of a user's organizations, the shortest configured duration across them applies. Disabling the setting does not retroactively extend sessions already scheduled to expire.",
shortValue: 'Partial: Enterprise/Console only, 1/7/14/28-day absolute cap',
shortValue: 'Yes: Enterprise/Console only, 1/7/14/28-day absolute cap',
confidence: 'verified',
sources: [
{
Expand Down
6 changes: 3 additions & 3 deletions apps/sim/lib/compare/data/competitors/crewai.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -656,10 +656,10 @@ export const crewaiProfile: CompetitorProfile = {
},
codeSandboxRuntime: {
value:
'Partial: CrewAI no longer ships a sandbox of its own. The Docker-backed CodeInterpreterTool was removed from crewai-tools in April 2026 and the docs now direct users to a third-party sandbox service, so the execution environment is configured in that provider. What remains first-party is AMP deployment: a deployed crew installs whatever the project declares in pyproject.toml/uv.lock, including private-registry packages.',
'Yes: at the package layer on CrewAI AMP, where a deployed crew installs whatever the project declares in pyproject.toml/uv.lock, including private-registry packages. There is no first-party sandbox to configure: the Docker-backed CodeInterpreterTool was removed from crewai-tools in April 2026 and the docs now direct users to a third-party sandbox service, so image and resource configuration happen in that provider instead.',
detail:
"The CodeInterpreterTool page now carries a deprecation warning stating that the tool 'has been removed from crewai-tools' and that the allow_code_execution and code_execution_mode parameters on Agent are also deprecated, directing users to a dedicated sandbox service (E2B or Modal) instead; CERT/CC VU#221883 records the same vendor statement, that the tool including its Docker sandbox and its restricted-Python fallback was removed in response to code-execution vulnerabilities. In its place crewai-tools ships wrappers for third-party sandboxes (e2b_sandbox_tool, daytona_sandbox_tool), so image, package, and resource configuration follow that provider's model and account rather than CrewAI's. Separately, a crew deployed to CrewAI AMP is built from the repository's own pyproject.toml with a required uv.lock, so the deployed runtime contains the packages the developer declared, including packages from a private registry configured through [[tool.uv.index]] plus UV_INDEX_*_USERNAME/PASSWORD environment variables. Outside AMP, CrewAI is a Python library with no sandbox of its own: crew code runs in whatever interpreter and virtualenv the developer starts it in, so the dependency set is fully theirs to control and equally unisolated.",
shortValue: 'Partial, AMP deploy dependencies; code sandbox delegated to E2B/Daytona',
shortValue: 'Yes, AMP deploy dependencies only; code sandbox delegated to E2B/Daytona',
confidence: 'verified',
sources: [
{
Expand Down Expand Up @@ -937,7 +937,7 @@ export const crewaiProfile: CompetitorProfile = {
},
sessionPolicy: {
value:
"Not publicly documented: no admin-configurable session lifetime or idle timeout appears in CrewAI AMP's public docs; the SSO, RBAC, and self-hosted configuration pages name no session control",
'No: no admin-configurable session lifetime or idle timeout is documented for CrewAI AMP on any deployment mode; the SSO, RBAC, and self-hosted Helm configuration pages name no session control, and CrewAI does not document where session length is set. The identity providers it lists (WorkOS, Microsoft Entra ID, Okta, Auth0, Keycloak) are third-party, so any timeout configured there comes from the IdP rather than from CrewAI. The open-source framework has no user accounts or sign-in at all.',
detail:
"CrewAI's platform SSO page documents WorkOS (the SaaS default), Microsoft Entra ID, Okta, Auth0, and Keycloak as identity providers and delegates MFA enforcement to the IdP, but names no session-lifetime, absolute-cap, or inactivity-timeout setting; the RBAC permission matrix likewise covers default_settings and organization_settings without a session control. The self-hosted Helm chart's WorkOS variables (WORKOS_CLIENT_ID, WORKOS_AUTHKIT_DOMAIN, WORKOS_COOKIE_PASSWORD, WORKOS_API_KEY) document credentials and cookie encryption but no lifetime or expiry setting. CrewAI does not document where session length is controlled, and no session-duration behavior is stated either way. The open-source framework has no user accounts or sign-in at all, so the question does not apply to it.",
shortValue: 'No documented session-lifetime or idle-timeout setting',
Expand Down
2 changes: 1 addition & 1 deletion apps/sim/lib/compare/data/competitors/dust.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -928,7 +928,7 @@ export const dustProfile: CompetitorProfile = {
},
sessionPolicy: {
value:
'Not publicly documented: Dust documents SAML SSO with workspace-wide enforcement and SCIM provisioning on the Enterprise plan, but no admin-configurable absolute session lifetime or idle timeout, and no fixed session length is published either',
'No: no admin-configurable absolute session lifetime or idle timeout is documented anywhere, and no fixed session length is published either. Dust documents SAML SSO with workspace-wide enforcement and SCIM provisioning on the Enterprise plan, but enforcing SSO governs which login methods are accepted rather than how long a signed-in session lasts, so session length in practice follows whatever the upstream third-party identity provider enforces at re-authentication, which is not a Dust-owned policy',
detail:
"Dust's workspace governance docs describe the admin surface under Admin > People & Security as managing security settings, user access, identity verification, and provisioning; neither those pages nor the SSO/SAML pages describe a session-duration, idle-timeout, or forced re-authentication setting. Enforcing SSO restricts which login methods are accepted (users can no longer sign in with social accounts) rather than how long a signed-in session lasts, so session length in practice follows whatever the upstream identity provider enforces at re-authentication.",
shortValue: 'No documented session-lifetime or idle-timeout setting',
Expand Down
9 changes: 5 additions & 4 deletions apps/sim/lib/compare/data/competitors/flowise.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -536,10 +536,10 @@ export const flowiseProfile: CompetitorProfile = {
},
codeSandboxRuntime: {
value:
'Partial: self-hosted only and at the image level. The runtime available to Custom JS Function and Custom Tool code is controlled by server environment variables: TOOL_FUNCTION_BUILTIN_DEP (which NodeJS built-in modules code may require, set to crypto,fs in the shipped .env.example and documented as accepting * for all builtins), TOOL_FUNCTION_EXTERNAL_DEP (which npm packages are importable, set to moment,lodash,pg,mysql2,mongodb,ioredis,redis,typeorm,@zilliz/milvus2-sdk-node in the same file), and ALLOW_BUILTIN_DEP, which the docs describe as allowing the project dependencies already bundled with Flowise to be used, illustrated with cheerio and typeorm. Adding an npm package that is not already bundled requires adding it to packages/components, rebuilding Flowise, and restarting the server, so it is a redeploy of your own image rather than a per-step dependency declaration; there is no runtime npm install inside the code node, and no documented way to declare OS-level system packages or preinstalled CLI binaries. Flowise does not document any way for users of the managed Flowise Cloud product to set these server environment variables.',
'Yes: on self-hosted deployments only, and at the image level. The runtime available to Custom JS Function and Custom Tool code is controlled by server environment variables: TOOL_FUNCTION_BUILTIN_DEP (which NodeJS built-in modules code may require, set to crypto,fs in the shipped .env.example and documented as accepting * for all builtins), TOOL_FUNCTION_EXTERNAL_DEP (which npm packages are importable, set to moment,lodash,pg,mysql2,mongodb,ioredis,redis,typeorm,@zilliz/milvus2-sdk-node in the same file), and ALLOW_BUILTIN_DEP, which the docs describe as allowing the project dependencies already bundled with Flowise to be used, illustrated with cheerio and typeorm. Adding an npm package that is not already bundled requires adding it to packages/components, rebuilding Flowise, and restarting the server, so it is a redeploy of your own image rather than a per-step dependency declaration; there is no runtime npm install inside the code node, and no documented way to declare OS-level system packages or preinstalled CLI binaries. Flowise does not document any way for users of the managed Flowise Cloud product to set these server environment variables.',
detail:
'The Custom Tool docs describe the workflow explicitly: add the package under packages/components with pnpm add, run pnpm build, "then, add the imported libraries to TOOL_FUNCTION_EXTERNAL_DEP environment variable", then restart the application. TOOL_FUNCTION_BUILTIN_DEP is additive rather than a replacement: packages/components/src/utils.ts concatenates it onto a hardcoded defaultAllowBuiltInDep list (assert, buffer, crypto, events, path, querystring, timers, url, zlib), so the shipped crypto,fs value widens that base set rather than defining it; TOOL_FUNCTION_EXTERNAL_DEP is additive in the same way, layered onto a defaultAllowExternalDependencies list of axios and node-fetch. Python is not a supported code-step runtime at all, so the configurable surface is Node-only.',
shortValue: 'Self-hosted only: env-var allowlists plus a rebuild; no Cloud control',
shortValue: 'Yes, self-hosted only: env-var allowlists plus a rebuild; no Cloud control',
confidence: 'verified',
sources: [
{
Expand Down Expand Up @@ -789,10 +789,11 @@ export const flowiseProfile: CompetitorProfile = {
},
sessionPolicy: {
value:
'Partial: self-hosted server environment variables rather than an admin-console setting. JWT_TOKEN_EXPIRY_IN_MINUTES sets the access-token lifetime and JWT_REFRESH_TOKEN_EXPIRY_IN_MINUTES sets the refresh-token lifetime, which together bound how long a sign-in survives, and EXPIRE_AUTH_TOKENS_ON_RESTART invalidates all tokens on server restart. No idle/inactivity timeout is documented, and Flowise does not document any equivalent control for admins of the managed Flowise Cloud product.',
'Yes: on self-hosted deployments only, through server environment variables rather than an admin-console setting. JWT_TOKEN_EXPIRY_IN_MINUTES sets the access-token lifetime and JWT_REFRESH_TOKEN_EXPIRY_IN_MINUTES sets the refresh-token lifetime, which together bound how long a sign-in survives, and EXPIRE_AUTH_TOKENS_ON_RESTART invalidates all tokens on server restart. No idle/inactivity timeout is documented, and Flowise does not document any equivalent control for admins of the managed Flowise Cloud product.',
detail:
'The shipped .env.example sets JWT_TOKEN_EXPIRY_IN_MINUTES=360 and JWT_REFRESH_TOKEN_EXPIRY_IN_MINUTES=43200 (30 days); the Application authorization docs page instead cites defaults of 60 minutes and 129,600 minutes (90 days), so the documented default differs from the shipped one. Because these are process environment variables, changing them is a deploy-level action for the whole instance, not a per-organization policy an account admin can set from the Workspaces UI.',
shortValue: 'Self-hosted JWT expiry env vars; no idle timeout, no documented Cloud control',
shortValue:
'Yes, self-hosted JWT expiry env vars; no idle timeout, no documented Cloud control',
confidence: 'verified',
sources: [
{
Expand Down
4 changes: 2 additions & 2 deletions apps/sim/lib/compare/data/competitors/gumloop.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -980,10 +980,10 @@ export const gumloopProfile: CompetitorProfile = {
},
sessionPolicy: {
value:
'Not publicly documented: no admin-configurable session lifetime or idle timeout appears in Gumloop\'s public documentation. The only primary-source statement is a single bullet on the Enterprise SSO page listing "Session Management: Configurable session timeouts and secure token handling." No admin setting name, configuration steps, default session lifetime, idle-timeout value, or absolute-cap value is published anywhere in Gumloop\'s docs, security page, or trust center.',
'No: no admin-configurable session lifetime or idle timeout appears in Gumloop\'s public documentation. The only primary-source statement is a single bullet on the Enterprise SSO page listing "Session Management: Configurable session timeouts and secure token handling." No admin setting name, configuration steps, default session lifetime, idle-timeout value, or absolute-cap value is published anywhere in Gumloop\'s docs, security page, or trust center, so a buyer has no documented control to rely on.',
detail:
Comment on lines 982 to 984

@greptile-apps greptile-apps Bot Aug 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Gumloop session verdict contradicts source

When the comparison renders Gumloop’s session policy, the changed No: prefix classifies it as unsupported even though Gumloop’s cited Enterprise SSO documentation states that configurable session timeouts are supported, causing a false negative in the table and structured comparison data.

"The bullet sits in a Security & Compliance list introduced as \"Gumloop's SSO implementation follows industry security standards\", alongside SOC 2 Type II certification, SAML 2.0 and TLS 1.3 entries that are compliance assurances rather than admin-configurable settings, so it does not establish a customer-facing control. The same page documents its actual admin controls in detail, including per-direction SCIM mapping-table toggles, name-based mapping mode, and named audit events, but describes no session setting; it gates SAML and SCIM settings to the Admin organization role and an Enterprise subscription, so any such control would be Enterprise-tier. Gumloop's public security page and trust center make no mention of session lifetime, idle timeout, or forced re-authentication. Because SAML SSO is available, organizations can also inherit re-authentication frequency from their upstream IdP, but that is the IdP's policy rather than a Gumloop-enforced one.",
shortValue: 'Single SSO-page bullet cites configurable timeouts; no setting documented',
shortValue: 'No: no documented session lifetime or idle-timeout setting',
confidence: 'estimated',
sources: [
{
Expand Down
4 changes: 2 additions & 2 deletions apps/sim/lib/compare/data/competitors/langchain.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -935,10 +935,10 @@ export const langchainProfile: CompetitorProfile = {
},
sessionPolicy: {
value:
'Partial: self-hosted LangSmith only. An operator sets the maximum session length via the OAUTH_SESSION_MAX_SEC environment variable when using OAuth 2.0/SSO, or BASIC_AUTH_JWT_EXPIRATION_SECONDS when using basic authentication, both defaulting to 28800 seconds (8 hours). No equivalent admin-configurable session control is documented for LangSmith Cloud.',
'Yes: on self-hosted LangSmith deployments only, where an operator sets the maximum session length via the OAUTH_SESSION_MAX_SEC environment variable when using OAuth 2.0/SSO, or BASIC_AUTH_JWT_EXPIRATION_SECONDS when using basic authentication, both defaulting to 28800 seconds (8 hours). No equivalent admin-configurable session control is documented for LangSmith Cloud.',
detail:
'Both settings are an absolute cap on session lifetime from sign-in; no separate inactivity/idle timeout is documented. On the cloud product, LangChain support states that after a SAML assertion is validated the session token (a JWT) is issued by LangSmith\'s own auth layer rather than the IdP, and that this "is by design and is not configurable", so an upstream IdP session policy does not govern the resulting LangSmith session either.',
shortValue: 'Self-hosted env vars (8h default); not configurable on cloud',
shortValue: 'Yes, self-hosted env vars (8h default); not configurable on cloud',
confidence: 'verified',
sources: [
{
Expand Down
Loading
Loading