Building a Configurable Task Engine: Versioning, Event-Driven Architecture, and Migration Strategies

Building a flexible task management system for complex workflows requires careful architectural decisions around business rules, data modeling, and migration strategies. This article explores the design of a production task engine for mortgage loan processing, focusing on key architectural challenges: creating a configurable rules system that non-technical users can manage, implementing asynchronous task generation with intelligent diffing, ensuring task versioning prevents historical confusion, and executing a zero-downtime migration using the strangler fig pattern.

The Problem Space

Task management in mortgage processing needs to adapt dynamically to diverse workflows. Different loan types, property configurations, and business requirements generate different task sets. A purchase loan for a single-family home requires different documentation than a cash-out refinance on an investment property.

The requirements included:

• User-configurable business rules: Business users need to define task generation logic based on loan attributes without developer intervention
• Borrower-specific tasks: Each borrower on a loan gets their own task set, with granular targeting options
• Rich task types: Support for document uploads, plain text responses, and automated verification services (VOA, VOI, VOE)
• Collaboration features: Borrowers and loan officers need threaded communication on specific tasks
• Flexible task management: Support for rule-based automatic generation, manual imports, and ad-hoc task creation
• Task versioning: Template changes must not affect existing task instances
• Migration path: Replace an unreliable legacy system without downtime or data migration complexity

The challenge was building a system that balanced flexibility with maintainability while ensuring production reliability and data integrity.

System Architecture

Data Model

The task system is built on three core entities:

loanTasks (Task Templates, Ad-hoc Tasks, Rule Tasks)

• Task configuration and metadata
• Type specification (document, text, verification)
• Due date offset from generation
• Borrower targeting rules (specific borrowers or all)
• Auto-completion configuration
• Auto-generation diff configuration
• Sort weight for ordering
• Version tracking for template changes

loanUserTasks (Generated Task Instances)

• Ties tasks to specific borrowers on specific loans
• Tracks lifecycle status (Outstanding, Pending, Completed, Rejected)
• Stores completion metadata and timestamps
• Maintains event log for key actions
• References specific template version at generation time

loanUserTaskComments

• Threaded conversations on specific tasks
• Enables borrower-loan officer collaboration

This separation between templates and instances allows rule modifications without affecting in-flight tasks while maintaining clear audit trails of task generation.

Task Versioning

Task templates evolve over time as business requirements change. Without versioning, template modifications would retroactively affect all existing task instances, creating confusion and data integrity issues.

The Versioning Challenge:

Consider a task template named "Driver's License" used to generate tasks on hundreds of active loans. If an administrator changes the template name to "Government ID," all existing task instances would immediately reflect this change—even those where borrowers already uploaded their driver's license. The mismatch between the task name and uploaded content creates confusion for loan officers reviewing completed work.

Similarly, other template changes create problems:

• Folder destination changes: Previously uploaded documents appear in unexpected locations
• Auto-completion rule changes: Tasks completed under old rules might not meet new criteria
• Task type changes: Converting a text task to a document task invalidates existing text responses

The Versioning Solution:

Task instances capture a snapshot of template configuration at generation time:

• Immutable core properties: Task name, type, and description frozen at generation
• Template version binding: Each instance references the specific template version that created it
• Independent evolution: Template modifications only affect newly generated tasks
• Audit trail preservation: Historical tasks remain coherent and understandable

This approach ensures that:

• Completed tasks maintain their original context
• Uploaded documents align with task descriptions
• Audit trails make sense when reviewing loan history
• Template administrators can modify configurations without unintended consequences

Implementation:

When a task template is modified, the system creates a new version. Generated task instances store:

• Template ID and version number
• Snapshot of critical display fields (name, description, type)
• Reference to template version for additional metadata

New task generation always uses the latest template version, while existing instances remain bound to their original version. This prevents retroactive changes while allowing templates to evolve.

Business Rules Engine

The rules engine is the heart of the system, enabling business users to define task generation logic through a UI without code changes.

Rule Structure:

Each rule consists of configurable conditions and resulting task sets:

• Condition chains: Logical AND/OR combinations of loan attributes (e.g., "Loan Purpose = Refinance AND Property Type = Single Family")
• Custom field support: Rules can reference custom loan fields for advanced scenarios
• Maximum complexity: 10 conditions per rule to maintain simplicity and performance
• Task mapping: Each rule generates a specific set of tasks when conditions match

Rule Validation:

The system detects contradictory rules during configuration—for example, mutually exclusive conditions that would never match. This validation prevents common configuration mistakes and provides immediate feedback to administrators.

Task Templates:

Beyond rules, the system supports "Task Templates" (reusable task templates) that can be manually imported to any loan. This provides flexibility for one-off situations that don't fit standard rule patterns.

Task Generation Strategy

The system uses event-driven architecture for task generation, decoupling user actions from task creation to improve performance and reliability.

Primary Generation Path (Asynchronous):

When a borrower is added to a loan, the system publishes a UserAddedToLoan event that a MassTransit consumer processes asynchronously:

1. User addition completes immediately (registration, invitation acceptance, etc.)
2. Event published to message queue
3. Consumer evaluates user role and matching business rules against loan data
4. Tasks generated for the new borrower using current template versions
5. Task status changes logged via separate events

This async approach provides several benefits:

• Non-blocking user experience: Users aren't waiting for task generation during registration or onboarding flows
• Retry capability: Failed task generation can be retried without user impact
• Separation of concerns: User management operations are decoupled from business rule evaluation
• Performance: Complex rule evaluation doesn't block request threads

Secondary Generation Paths:

• Manual imports (synchronous): When loan officers manually import preliminary/underwriting conditions or task templates, generation happens synchronously since they're actively waiting for the result
• Automatic re-evaluation (asynchronous): When loans change and automatic mode is enabled, a LoanUpdated event triggers re-evaluation via a separate consumer to see if new tasks match based on the loan changes
• Ad-hoc tasks (synchronous): Direct task creation by loan officers completes immediately

Task Generation Triggers:

• Borrower addition: Event-driven task generation for new borrowers
• Configuration changes: On-demand diff or automatic re-evaluation depending on configuration

Configuration Change Handling:

When business rules change, we support two modes:

• Manual (default): Loan officers see a diff of newly matching tasks and selectively import them
• Automatic (opt-in): LoanUpdated events trigger re-evaluation, automatically adding newly matching tasks. This is configurable on a task basis.

The default manual mode gives users control over when new tasks appear, preventing unexpected changes to active workflows.

Task Lifecycle and Interactions

Tasks support rich functionality throughout their lifecycle:

Task Types:

• Document tasks: Integrated upload flow with configurable folder destinations
• Text tasks: Free-form responses for information collection
• Verification tasks: Borrowers can initiate VOA/VOI/VOE services directly, with async processing and notifications

Task Features:

• Auto-completion: Tasks automatically complete when borrowers provide required information
• Due dates: Configurable deadlines with filtering and sorting
• Comments: Threaded discussions between borrowers and loan officers
• Weights: Custom sort ordering for task presentation
• Selective targeting: Tasks can target specific borrowers or all borrowers on a loan

Event-Driven Status Tracking:

Task status changes and completions publish events to MassTransit consumers that handle:

• Status transition logging
• Notification aggregation (detailed below)
• External system sync for essential data
• Audit trail generation

Task Deduplication:

When multiple rules generate identical tasks for a borrower, the system automatically deduplicates to prevent redundant work.

Key Architectural Decisions

Task Versioning Prevents Historical Confusion

The decision to implement task versioning was driven by data integrity and user experience concerns. Without versioning, the system would have no memory of what tasks meant when they were created.

Why versioning matters:

In mortgage processing, loans can remain active for months. A task generated in January with specific requirements shouldn't change meaning in March when the template is updated for new loans. Loan officers reviewing completed work need to understand what was actually requested, not what the current template says.

Versioning also enables confident template evolution. Administrators can refine task names, update descriptions, or change auto-completion rules knowing existing loans won't be affected. This removes the fear of making improvements to templates.

Implementation tradeoffs:

The versioning approach increases storage (snapshots of template data) and adds complexity to queries (joining version information). However, these costs are minimal compared to the data integrity issues that would arise without versioning.

Versioning represents the right balance: templates can evolve naturally while preserving historical context.

Event-Driven Task Generation

Choosing asynchronous task generation over synchronous processing was a fundamental architectural decision that shaped the entire system.

Why async-first?

Consider the user journey: when a borrower registers or accepts an invitation, they're directed to their dashboard or prompted to fill out initial information. They're not immediately interacting with tasks. Blocking their registration flow to wait for potentially complex rule evaluation would degrade user experience without benefit.

The async approach also provides:

• Reliability: If rule evaluation fails (database timeout, external system unavailable), the event can be retried without the user experiencing an error
• Scalability: Task generation load is handled by dedicated consumers that can scale independently
• Flexibility: We can add additional processing (analytics, notifications, webhooks) to the same events without modifying user-facing code

When synchronous makes sense:

Manual imports and ad-hoc task creation remain synchronous because users are actively waiting for results. When a loan officer clicks "Import Conditions," they expect immediate feedback. The context differs from automated background generation.

This pragmatic mix—async by default, sync where appropriate—demonstrates that architectural patterns should serve user needs.

On-Demand Diff vs. Real-Time Re-evaluation

When business rules change, we faced a fundamental architectural question: should the system automatically regenerate tasks for all existing loans, or provide a diff that users can review?

The real-time approach would have meant:

• Background jobs re-evaluating all active loans after rule changes
• Automatically adding newly matching tasks
• Deciding whether to remove tasks that no longer match
• Complex conflict resolution for partially completed work

We chose on-demand diff instead:

• When accessing a loan's task page, the system calculates what current rules would generate
• Users see a diff between existing tasks and rule-generated tasks
• Manual mode (default): users selectively import new tasks
• Automatic mode (opt-in): new tasks added on loan updates via async event processing

Tradeoffs:

The on-demand approach trades perfect consistency for pragmatism:

Advantages:

• User control: No unexpected task changes in active workflows
• Simpler conflict resolution: Never automatically remove tasks, avoiding lost borrower work
• Performance: No expensive bulk re-evaluation across thousands of loans
• Predictable behavior: Task generation happens at defined trigger points

Disadvantages:

• Eventual consistency: Task sets may be slightly stale if rules change and loans aren't accessed
• User action required: Manual mode requires loan officers to import new tasks

In practice, business rules change infrequently once established. Most lenders modify rules during initial setup, then rarely change them. The complexity of real-time re-evaluation wasn't justified by the low frequency of rule changes in production.

Data Ownership and Decoupling

The legacy system stored task data in serialized JSON fields within the external LOS. This approach created fundamental problems:

• Reliability issues: External system outages blocked all task operations
• Limited querying: Serialized data prevented efficient filtering, sorting, and reporting
• Performance constraints: Every task operation required external API calls
• Inflexible schema: Adding task features required external system changes

Our approach: Own the data

We store all task data in our own database with strategic sync points to the external system:

• Task state and lifecycle: Fully owned by our system
• Business rule evaluation: Runs against local data
• User interactions: No external dependencies for core workflows
• Lightweight sync: Only essential data (document references, status counts) syncs to external systems

This separation of concerns gave us:

• Independence: Core task workflows function regardless of external system availability
• Performance: All queries against local database with proper indexing
• Flexibility: New features don't require external system changes
• Reliability: Reduced failure modes and simplified error handling

Notification Aggregation

Rather than sending notifications for every task event, we implement delayed aggregation:

• Task completions and comments trigger a 5-minute delay timer
• Additional events during the delay reset the timer
• After 5 minutes of inactivity, a single aggregated notification is sent

This prevents notification storms when borrowers complete multiple tasks rapidly while ensuring timely updates for single-task scenarios. Users receive meaningful updates without being overwhelmed.

Migration Strategy: Strangler Fig Pattern

Replacing the legacy task system presented a migration challenge. The old system stored data in an external system with an unreliable integration and inflexible data model. A traditional migration approach would have required:

• Extracting task data from serialized JSON
• Mapping legacy tasks to new schema
• Complex data transformation logic
• Risk of data loss or corruption
• Extended downtime during cutover

We chose the strangler fig pattern instead:

Phase 1: Dual Operation (90 days)

• New loans immediately used the new task engine
• Existing loans continued using the legacy system
• Both systems ran in parallel
• Minimal code changes to support dual modes
• No data migration required

Phase 2: Natural Expiration

• Most mortgage loans close or are abandoned within 90 days
• Business stakeholders chose 90 days as the cutover timeline
• Loans still active after 90 days could manually import their tasks if needed

Risk Assessment:

This approach carried risk—launching directly to all new loans without gradual rollout. However, the legacy system was unreliable enough that any risk from the new system was acceptable. The business priority was moving off the legacy system as quickly as possible.

Benefits of the Strangler Fig Approach:

• No bulk data migration: Avoided complex ETL from serialized JSON
• Immediate production validation: New system tested with real traffic from day one
• Simple code changes: Minimal branching logic to support dual modes
• Clear end date: Natural loan lifecycle provided automatic phase-out
• Low risk cutover: No "big bang" migration event

The use of the strangler fig pattern meant we never needed a bulk migration process. The system handled the transition gracefully based on actual usage patterns.

Production Considerations

Performance and Scalability

With all loans having tasks, the system needed efficient query patterns:

• Database indexing: Indexes on status, due date, user, and loan identifiers
• Pagination: Task lists paginated to handle loans with 50+ tasks
• Efficient rule evaluation: Rules evaluated against local data, not external API calls
• Query optimization: Filtered queries for common use cases (overdue tasks, pending documents)

Message Processing and Reliability

The event-driven architecture requires careful attention to message processing:

• Consumer idempotency: Task generation consumers handle duplicate events gracefully
• Retry policies: Failed message processing retries with exponential backoff
• Dead letter queues: Persistently failing messages move to DLQ for investigation
• Monitoring: Consumer lag and failure rates tracked for operational visibility

Data Integrity and Reconciliation

Background jobs ensure data consistency (LOS is the source of truth for loan data):

• Rule evaluation accuracy: Periodic reconciliation of loan data and custom field values
• External sync verification: Confirmation that essential syncs completed successfully
• Orphaned task detection: Identification of tasks without valid borrowers or loans

Administrators have manual sync capabilities for immediate reconciliation needs without waiting for background jobs.

Task Import Features

Beyond rule-based generation, the system supports flexible task imports:

• Condition imports: Pull underwriting and preliminary conditions from external systems, with selective import UI
• Template imports: Apply global task templates to specific loans
• Ad-hoc creation: Loan officers can create one-off tasks outside the rules engine

These features provide escape hatches for workflows that don't fit standard rule patterns.

Lessons Learned

Task Versioning is Non-Negotiable

Any system where templates evolve over time needs versioning. The cost of implementation (additional storage, query complexity) is trivial compared to the data integrity issues and user confusion that arise without it. Build versioning from day one.

Event-Driven Architecture Enables Scale and Reliability

Decoupling task generation from user actions via events provided performance benefits while enabling retry logic and independent scaling. The async-first approach with pragmatic synchronous exceptions demonstrates that architectural patterns should serve user needs.

Data Ownership Enables Flexibility

Decoupling from external systems as our database gave us control over performance, reliability, and feature velocity. Lightweight sync points provide necessary integration without tight coupling. This separation of concerns was the most impactful architectural decision.

On-Demand Beats Real-Time for Infrequent Changes

The complexity of real-time rule re-evaluation wasn't justified by how rarely rules change in production. On-demand diff with user control proved more pragmatic and gave users agency over their workflows.

Strangler Fig Eliminates Migration Risk

When replacing critical systems, running both in parallel eliminates migration complexity while providing production validation. Let actual usage patterns drive migration effort.

User Control Prevents Surprise

Automatic task generation sounds efficient, but users value predictability especially in mortgage. Manual mode as the default, with automatic as opt-in, respects user workflows and prevents unexpected changes during critical loan processing stages.

Notification Aggregation is Essential

Users don't want individual notifications for every action. Delayed aggregation with reset timers provides timely updates without overwhelming users with notification storms.

Conclusion

Building a configurable task engine requires balancing flexibility with simplicity, feature richness with maintainability, and autonomy with integration. The architectural decisions around task versioning, event-driven processing, data ownership, and migration strategy shaped a system that empowers business users while remaining maintainable for engineers.

The key insights:

• Task versioning preserves context: Templates evolve without affecting historical data
• Event-driven architecture enables reliability: Async-first with pragmatic sync exceptions
• Own your core data: Don't treat external systems as your database
• On-demand beats real-time for infrequent configuration changes
• User control prevents surprise: Automatic isn't always better
• Strangler fig: Eliminates migration risk
• Notification aggregation respects user attention: Batch related events