Issue: Loss of Important Fields During Source Configuration Retrieval

Problem Statement

While retrieving source information, we have encountered an issue where the expected source configuration dataclass (e.g., SourceSalesforce) is not being correctly returned. Instead, the configuration is returned as SourceAirbyte, which causes unintended field removal.

Breakdown of the Issue

1. Expected Behavior

• When we create a source configuration using a specific dataclass (e.g., SourceSalesforce), we expect to retrieve it in the same structured format.
• This ensures that all necessary fields (such as authentication details, API settings, etc.) are retained, allowing us to update or modify the source configuration seamlessly.

2. Observed Behavior

• Instead of returning the original dataclass (SourceSalesforce), the system is returning a more generic SourceAirbyte class.
• The SourceAirbyte class does not retain all the fields of SourceSalesforce.
• As a result, fields specific to SourceSalesforce are missing upon retrieval.

3. Impact of the Issue

• Field Removal: Since SourceAirbyte does not have the additional fields defined in SourceSalesforce, these fields are getting removed when retrieving and updating the configuration.
• Data Loss: Important configuration settings required for proper functionality are lost.
• Update Failures: When trying to modify the source configuration, required fields are missing, leading to issues when saving or syncing data.
• Unexpected Behavior: The system may function incorrectly due to missing parameters that were initially set.

Potential Causes

• Incorrect Serialization/Deserialization: If the system is serializing the source configuration without considering its original dataclass, it might be defaulting to SourceAirbyte, stripping out subclass-specific fields.
• Improper Type Handling: The retrieved data might be cast into a generic base class (SourceAirbyte) instead of preserving the specific subclass (SourceSalesforce).
• Schema Enforcement Issue: If the system has predefined schemas for different sources but defaults to SourceAirbyte during retrieval, this could lead to loss of information.
• Incorrect Database/Storage Retrieval: The stored configuration might not be properly mapped to its specific dataclass when fetching from storage.