Deployment Manager¶

The Deployment Manager is your control center for running, scheduling, and monitoring data pipelines. It provides three deployment types: immediate execution, CRON scheduling, and API endpoints.

Complete tour of the Deployment Manager interface

Overview¶

The Deployment Manager offers:

• Immediate Pipeline Execution - Run pipelines on-demand
• CRON Scheduled Deployments - Automate pipeline execution with scheduling
• API Endpoint Deployments - Expose pipelines as HTTP endpoints
• Jobs/Executions Monitoring - Track pipeline runs and performance
• Version Management - Deploy specific pipeline versions
• Environment Support - Deploy to different environments (DEV, PROD, etc.)

Main Deployment Manager showing active deployments

Deployment Types¶

1. Execute Pipeline (Immediate)¶

Run a pipeline immediately for testing or ad-hoc data processing.

Immediate pipeline execution dialog

When to Use: - Testing new data steps - One-time data processing tasks - Quick data analysis - Debugging pipeline issues

Configuration: - Workflow: Select the pipeline to execute - Version: Choose specific version (defaults to latest) - Environment: Select target environment - Single Instance: Prevent multiple simultaneous runs

2. CRON Deployment (Scheduled)¶

Schedule pipelines to run automatically using CRON expressions.

Setting up a scheduled deployment

When to Use: - Daily data processing jobs - Weekly reports - Monthly aggregations - Regular data synchronization

Configuration: - Name: Descriptive deployment name - Description: Optional deployment description - Workflow: Pipeline to schedule - Version: Specific version to run - Environment: Target environment - Schedule: CRON expression (e.g., 0 9 * * * for daily at 9 AM) - Timezone: Organization timezone for scheduling - Single Instance: Prevent overlapping executions

Detailed CRON deployment configuration

Common CRON Patterns¶

3. API Deployment¶

Expose pipelines as secure HTTP endpoints that can be triggered by external systems.

Creating an API endpoint deployment

When to Use: - Webhook integrations - External system triggers - Real-time data processing - Custom application integration

Configuration: - Workflow: Pipeline to expose - Version: Specific version to deploy - Environment: Target environment - Organization Token: Required for authentication - Single Instance: Control concurrent executions

API endpoint configuration and authentication

Managing Deployments¶

Deployments Tab¶

The main Deployments tab shows all your configured deployments:

Active deployments with status and actions

Deployment Information: - Name & Description: Deployment identification - Pipeline: Associated workflow and version - Type: CRON schedule or API endpoint indicator - Status: Active/Inactive state - Next Execution: Scheduled run time (for CRON deployments) - Actions: Edit, activate/deactivate, delete

Deployment Actions¶

Available actions for each deployment

Available Actions: - Edit: Modify deployment configuration - Activate/Deactivate: Enable or disable deployment - Delete: Remove deployment permanently

Jobs/Executions Tab¶

Monitor all pipeline executions across your deployments:

Monitoring pipeline executions and performance

Execution Information: - Job ID: Unique execution identifier - Pipeline: Which workflow was executed - Status: Running, Completed, Failed, Cancelled - Duration: Execution time (hours:minutes:seconds) - Started: Execution start time - Trigger: How the execution was initiated - Actions: View details, change status, retry

Detailed jobs execution list with filtering

Environment Management¶

Multiple Environments¶

data-conductor supports deploying to different environments:

Selecting deployment environment

Common Environment Types: - DEV: Development and testing - STAGING: Pre-production validation - PROD: Production data processing - TEST: Isolated testing environment

Environment Configuration¶

Each environment can have: - Different database connections - Separate configuration values - Isolated execution resources - Environment-specific variables

Version Management¶

Pipeline Versions¶

Deploy specific versions of your pipelines:

Selecting pipeline version for deployment

Version Benefits: - Stability: Deploy tested versions to production - Rollback: Easy reversion to previous versions - Testing: Test new versions in development - Consistency: Ensure same code runs across environments

Version Deployment Strategy¶

Recommended Approach: 1. Development: Use latest version for testing 2. Staging: Deploy specific stable versions 3. Production: Only deploy thoroughly tested versions

Monitoring and Troubleshooting¶

Execution Monitoring¶

Real-time execution monitoring and alerts

Monitor Key Metrics: - Execution Time: Performance trends - Success Rate: Reliability metrics - Error Patterns: Common failure points - Resource Usage: System performance

Common Issues and Solutions¶

CRON Not Running

Issue: Scheduled deployment not executing
Solutions:
1. Check deployment is active
2. Verify CRON expression syntax
3. Confirm timezone settings
4. Check system logs for errors

API Endpoint Not Responding

Issue: API deployment returning errors
Solutions:
1. Verify organization token is valid
2. Check API endpoint URL
3. Confirm payload format
4. Review authentication headers

Long Execution Times

Issue: Pipelines taking too long to complete
Solutions:
1. Optimize SQL queries in data steps
2. Add LIMIT clauses for testing
3. Check database performance
4. Consider data partitioning

Permission Errors

Issue: Access denied during execution
Solutions:
1. Verify database integration credentials
2. Check user permissions
3. Confirm IP address is trusted
4. Review organization token permissions

Best Practices¶

CRON Deployment Best Practices¶

Scheduling Guidelines: - Avoid peak business hours for heavy processing - Stagger multiple deployments to prevent resource conflicts - Use single instance to prevent overlapping runs - Set appropriate timeouts for long-running processes

Example Staggered Schedule:

ETL Pipeline A:     0 1 * * *  (1:00 AM)
ETL Pipeline B:     0 2 * * *  (2:00 AM)
Report Pipeline:    0 3 * * *  (3:00 AM)
Cleanup Pipeline:   0 4 * * *  (4:00 AM)

API Deployment Best Practices¶

Security: - Use organization tokens for authentication - Implement rate limiting on client side - Monitor API usage patterns - Rotate tokens regularly

Performance: - Design for idempotency - Handle timeouts gracefully - Implement retry logic with exponential backoff - Monitor response times

Monitoring Best Practices¶

Alerting Strategy: - Set up alerts for failed executions - Monitor execution time trends - Track resource usage patterns - Alert on unusual activity

Logging: - Capture detailed execution logs - Log input parameters and outputs - Track performance metrics - Maintain audit trails

Advanced Features¶

Single Instance Protection¶

Prevents multiple executions of the same pipeline:

Single instance protection configuration

Benefits: - Prevents resource conflicts - Ensures data consistency - Avoids duplicate processing - Maintains system stability

Execution Retry Logic¶

Configure automatic retries for failed executions:

Retry Configuration: - Maximum retry attempts - Retry delay intervals - Failure conditions for retry - Escalation procedures

Custom Variables per Environment¶

Set different variable values for each environment:

DEV Environment:
- DATABASE_HOST: dev-db.company.com
- BATCH_SIZE: 100

PROD Environment:
- DATABASE_HOST: prod-db.company.com
- BATCH_SIZE: 10000

Next Steps¶

Now that you understand the Deployment Manager:

• Learn about Pipeline Variables for dynamic configurations
• Set up IP Security to secure your deployments
• Explore API Integrations for external system connections
• Check Troubleshooting Guide for common issues

Ready to secure your organization? Continue to Organization Settings!