Getting Started with data-conductor¶

This guide will help you set up data-conductor and create your first data pipeline in just a few minutes.

Quick walkthrough of the setup process

Prerequisites¶

Before you begin, make sure you have:

• [ ] Access to a data-conductor instance
• [ ] Admin permissions for your organization
• [ ] Database connection details (if using SQL data steps)

Step 1: Organization Setup¶

When you first log into data-conductor, you'll need to configure your organization settings.

Organization settings page

Setting Up Your Organization¶

1. Navigate to Organization in the main menu
2. Fill in your organization details:
3. Organization name
4. Description
5. Timezone (used for CRON scheduling)

Configuring basic organization information

Creating API Tokens¶

API tokens are required for certain features like API deployments.

1. Go to the API Tokens tab
2. Click Create New Token
3. Provide a name and description
4. Save the token securely (you won't see it again)

Creating your first API token

Step 2: Database Integration Setup¶

If you plan to use SQL data steps, you'll need to set up database integrations.

Setting up your first database connection

Adding SQL Integrations¶

1. Navigate to Integrations (this may be in your admin panel)
2. Click Add SQL Integration
3. Configure your database connection:
4. Name: A friendly name for your database
5. Type: PostgreSQL, MySQL, etc.
6. Host: Database server address
7. Port: Database port
8. Database: Database name
9. Credentials: Username and password

Configuring a database connection

Testing Your Connection¶

Always test your database connection:

1. Click Test Connection
2. Verify the connection succeeds
3. Save your integration

Successful database connection test

Step 3: IP Security (Optional but Recommended)¶

For enhanced security, configure trusted IP addresses.

Managing trusted IP addresses

Adding Trusted IPs¶

1. Go to Organization → Trusted IPs tab
2. Click Add Trusted IP
3. Enter IP addresses or CIDR ranges
4. Add descriptions for easy management

Step 4: Your First Data Step¶

Now let's create a simple data step to test everything works.

Step-by-step creation of your first data step

Creating a SQL Data Step¶

1. Navigate to Data Builder
2. Click + New → SQL Data Step
3. Fill in the basic information:
4. Name: "Test Data Step"
5. Description: "My first data step"
6. Select your database integration
7. Write a simple SQL query:

SELECT
    'Hello data-conductor!' as message,
    NOW() as timestamp

Writing your first SQL query

Testing Your Query¶

1. Click the Execute button
2. Review the results in the Results tab
3. Verify your data looks correct

Successful query execution

Saving Your Data Step¶

1. Click Save when you're satisfied
2. Your data step is now ready for deployment

Step 5: Your First Deployment¶

Let's deploy your data step so it can run automatically.

Deploying your first pipeline

Creating a CRON Deployment¶

1. Navigate to Deployment Manager
2. Click + New Deployment → Attach Pipeline to CRON
3. Configure your deployment:
4. Name: "Daily Test Run"
5. Description: "Runs my test query daily"
6. Workflow: Select your test data step
7. Version: Use latest version
8. Environment: PROD
9. Schedule: 0 9 * * * (daily at 9 AM)

Configuring a scheduled deployment

Monitoring Your Deployment¶

1. Save your deployment
2. Check the Jobs/Executions tab to monitor runs
3. View execution logs and results

Monitoring pipeline executions

Next Steps¶

Congratulations! You've successfully set up data-conductor and created your first pipeline. Here's what to explore next:

Immediate Next Steps¶

• Create more complex data steps with variables and transformations
• Set up API endpoints to trigger pipelines on-demand
• Explore pipeline variables for dynamic queries

Advanced Features¶

• IP Security - Secure your instance
• API Integrations - Connect external systems
• Troubleshooting - Common issues and solutions

Common Issues¶

Connection Problems¶

If you're having trouble connecting to your database:

1. Check your firewall settings
2. Verify credentials are correct
3. Ensure the database server is accessible from data-conductor
4. Test with a simple tool like telnet first

Permission Issues¶

If you can't access certain features:

1. Verify your user role and permissions
2. Check with your organization administrator
3. Ensure your IP address is in the trusted list

Execution Failures¶

If your pipelines aren't running:

1. Check the execution logs in Jobs/Executions
2. Verify your SQL syntax
3. Ensure your database integration is still working
4. Check for timeout issues with long-running queries

Need Help?¶

• FAQ - Common questions and answers
• Troubleshooting - Detailed problem-solving guide
• Contact Support - Reach out to your administrator

Ready to build more complex pipelines? Continue to Your First Pipeline!