Skip to content
My Docs

59_Docker_For_Reproducible_Environments

Category: AI & Data Science Tools
Type: AI/ML Tool or Library
Generated on: 2025-08-26 11:10:43
For: Data Science, Machine Learning & Technical Interviews

Docker for Reproducible Environments (AI Tools & Libraries)

Section titled “Docker for Reproducible Environments (AI Tools & Libraries)”

This cheatsheet provides a comprehensive guide to using Docker for creating reproducible environments for AI and Data Science projects. It covers installation, core features, practical examples, and advanced techniques, tailored for data scientists and ML engineers.

1. Tool/Library Overview - Docker

  • What it is: Docker is a platform for developing, shipping, and running applications inside containers. A container packages an application and all its dependencies, ensuring consistent execution across different environments.
  • Main Use Cases in AI/ML:
    • Reproducibility: Guarantees that code runs the same way on different machines (development, testing, production).
    • Dependency Management: Isolates project dependencies, avoiding conflicts between different projects.
    • Scalability: Easily scales applications by running multiple containers.
    • Deployment: Simplifies deployment to cloud platforms or on-premise servers.
    • Collaboration: Allows sharing of environments with colleagues.
    • Version Control: Container images can be versioned, enabling rollbacks and experimentation.

2. Installation & Setup

3. Core Features & API

  • Dockerfile: A text file containing instructions for building a Docker image.
  • Docker Image: A read-only template used to create Docker containers.
  • Docker Container: A running instance of a Docker image.
  • Docker Hub: A registry for storing and sharing Docker images.
  • Docker Compose: A tool for defining and running multi-container Docker applications.

Key Docker Commands:

CommandDescriptionExample
docker buildBuilds a Docker image from a Dockerfile.docker build -t my-image:latest .
docker runRuns a Docker container from an image.docker run -it --rm my-image:latest bash
docker psLists running containers.docker ps
docker stopStops a running container.docker stop <container_id>
docker rmRemoves a stopped container.docker rm <container_id>
docker imagesLists available Docker images.docker images
docker rmiRemoves a Docker image.docker rmi <image_id>
docker pullPulls an image from a registry (e.g., Docker Hub).docker pull ubuntu:latest
docker pushPushes an image to a registry.docker push my-username/my-image:latest
docker execExecutes a command inside a running container.docker exec -it <container_id> bash
docker logsShows the logs of a container.docker logs <container_id>
docker compose upBuilds, (re)creates, starts, and attaches to containers for a service.docker compose up -d
docker compose downStops and removes containers, networks, volumes, and images created by up.docker compose down

4. Practical Examples

Example 1: Creating a Simple Python Environment

  • Dockerfile:

    # Use an official Python runtime as a parent image
    FROM python:3.9-slim-buster
    

    # Set the working directory to /app
    WORKDIR /app
    

    # Copy the current directory contents into the container at /app
    COPY . /app
    

    # Install any needed packages specified in requirements.txt
    RUN pip install --no-cache-dir -r requirements.txt
    

    # Make port 8000 available to the world outside this container
    EXPOSE 8000
    

    # Define environment variable
    ENV NAME World
    

    # Run app.py when the container launches
    CMD ["python", "app.py"]

  • requirements.txt:

    Flask==2.0.1

  • app.py:

    from flask import Flask
    import os
    app = Flask(__name__)
    

    @app.route("/")
    def hello():
        name = os.environ.get('NAME', "World")
        return "Hello " + name + "!"
    

    if __name__ == "__main__":
        app.run(debug=True, host='0.0.0.0', port=8000)

  • Build the image:

    Terminal window
    docker build -t python-app:latest .

  • Run the container:

    Terminal window
    docker run -p 8000:8000 python-app:latest
    • Expected Output: The Flask application starts and listens on port 8000. Navigate to http://localhost:8000 in your browser to see “Hello World!”.
    • Verification: If you set ENV NAME DataScientist in the Dockerfile or when running the container with -e NAME=DataScientist, you will see “Hello DataScientist!”.

Example 2: TensorFlow Environment with GPU Support

  • Dockerfile:

    FROM tensorflow/tensorflow:latest-gpu
    

    WORKDIR /app
    

    COPY . /app
    

    RUN pip install --no-cache-dir -r requirements.txt
    

    CMD ["python", "train.py"]

  • requirements.txt:

    numpy
    pandas
    scikit-learn

  • train.py (Simplified example):

    import tensorflow as tf
    import numpy as np
    

    # Generate some dummy data
    X = np.random.rand(100, 10)
    y = np.random.randint(0, 2, 100)
    

    # Create a simple model
    model = tf.keras.models.Sequential([
        tf.keras.layers.Dense(16, activation='relu', input_shape=(10,)),
        tf.keras.layers.Dense(1, activation='sigmoid')
    ])
    

    model.compile(optimizer='adam',
                  loss='binary_crossentropy',
                  metrics=['accuracy'])
    

    # Train the model
    model.fit(X, y, epochs=10)
    

    print("Training complete!")

  • Build the image:

    Terminal window
    docker build -t tensorflow-gpu-env:latest .

  • Run the container (with GPU support):

    Terminal window
    docker run --gpus all -v $(pwd):/app tensorflow-gpu-env:latest

Example 3: Using Docker Compose for a Data Science Pipeline (Simplified)

  • docker-compose.yml:

    version: "3.9"
    services:
      data_ingestion:
        build: ./data_ingestion
        volumes:
          - ./data:/data
        depends_on:
          - database
        environment:
          DATABASE_URL: postgresql://user:password@database:5432/mydatabase
    

      model_training:
        build: ./model_training
        volumes:
          - ./models:/models
        depends_on:
          - data_ingestion
        environment:
          DATABASE_URL: postgresql://user:password@database:5432/mydatabase
    

      database:
        image: postgres:13
        environment:
          POSTGRES_USER: user
          POSTGRES_PASSWORD: password
          POSTGRES_DB: mydatabase
        ports:
          - "5432:5432"
        volumes:
          - db_data:/var/lib/postgresql/data
    

    volumes:
      db_data:

  • Directory Structure:

    .
    ├── data
    │   └── raw_data.csv
    ├── data_ingestion
    │   └── Dockerfile
    │   └── ingestion.py
    │   └── requirements.txt
    ├── model_training
    │   └── Dockerfile
    │   └── train.py
    │   └── requirements.txt
    ├── docker-compose.yml
    └── models

  • data_ingestion/Dockerfile:

    FROM python:3.9-slim-buster
    WORKDIR /app
    COPY . /app
    RUN pip install --no-cache-dir -r requirements.txt
    CMD ["python", "ingestion.py"]

  • data_ingestion/ingestion.py:

    import pandas as pd
    import os
    import psycopg2
    

    DATABASE_URL = os.environ.get("DATABASE_URL")
    

    try:
        conn = psycopg2.connect(DATABASE_URL)
        print("Connected to PostgreSQL database")
    except psycopg2.Error as e:
        print(f"Error connecting to PostgreSQL: {e}")
        exit(1)
    

    # Create a table (example)
    cursor = conn.cursor()
    cursor.execute("""
        CREATE TABLE IF NOT EXISTS raw_data (
            id SERIAL PRIMARY KEY,
            value TEXT
        );
    """)
    conn.commit()
    

    # Ingest data (example)
    df = pd.read_csv("/data/raw_data.csv")
    for index, row in df.iterrows():
        cursor.execute("INSERT INTO raw_data (value) VALUES (%s)", (row['value'],))
    conn.commit()
    

    cursor.close()
    conn.close()
    print("Data ingestion complete")

  • data_ingestion/requirements.txt:

    pandas
    psycopg2-binary

  • model_training/Dockerfile:

    FROM python:3.9-slim-buster
    WORKDIR /app
    COPY . /app
    RUN pip install --no-cache-dir -r requirements.txt
    CMD ["python", "train.py"]

  • model_training/train.py:

    import pandas as pd
    import os
    import psycopg2
    from sklearn.linear_model import LogisticRegression
    import joblib
    

    DATABASE_URL = os.environ.get("DATABASE_URL")
    

    try:
        conn = psycopg2.connect(DATABASE_URL)
        print("Connected to PostgreSQL database")
    except psycopg2.Error as e:
        print(f"Error connecting to PostgreSQL: {e}")
        exit(1)
    

    cursor = conn.cursor()
    cursor.execute("SELECT value FROM raw_data")
    data = cursor.fetchall()
    cursor.close()
    conn.close()
    

    df = pd.DataFrame(data, columns=['value'])
    df['target'] = [0,1] * (len(df) // 2)
    

    model = LogisticRegression()
    model.fit(df[['value']].values, df['target'].values)
    

    joblib.dump(model, "/models/model.joblib")
    print("Model training complete and saved to /models/model.joblib")

  • model_training/requirements.txt:

    pandas
    psycopg2-binary
    scikit-learn
    joblib

  • Execute:

    Terminal window
    docker compose up --build
    • Explanation: This sets up a Postgres database, a data ingestion service that populates the database from a CSV file, and a model training service that trains a simple model on the ingested data. The docker-compose.yml file defines the dependencies between the services, ensuring that the database is running before the data ingestion service starts, and the data ingestion service is complete before the model training service starts.

5. Advanced Usage

  • Multi-Stage Builds: Reduce image size by using multiple FROM instructions in your Dockerfile. The first stage builds dependencies, and the second stage copies only the necessary artifacts to the final image.

    # Build stage
    FROM python:3.9-slim-buster AS builder
    WORKDIR /app
    COPY . /app
    RUN pip install --no-cache-dir -r requirements.txt
    

    # Final stage
    FROM python:3.9-slim-buster
    WORKDIR /app
    COPY --from=builder /app .
    CMD ["python", "app.py"]

  • Dockerignore File: Create a .dockerignore file to exclude unnecessary files and directories from being copied into the image (e.g., .git, __pycache__, venv).

    .git
    __pycache__
    venv
    *.pyc
    data/ # Example of excluding a data directory

  • Health Checks: Define health checks in your Dockerfile to monitor the health of your application.

    HEALTHCHECK --interval=30s --timeout=10s --retries=3 CMD curl -f http://localhost:8000/ || exit 1

  • Secrets Management: Use Docker secrets to securely manage sensitive information like API keys and database passwords.

    • Create a secret:
      Terminal window
      echo "mysecretpassword" | docker secret create db_password -
    • Use the secret in docker-compose.yml: 
      version: "3.9"
      services:
        database:
          image: postgres:13
          environment:
            POSTGRES_PASSWORD_FILE: /run/secrets/db_password
          secrets:
            - db_password
      secrets:
        db_password:
          external: true

6. Tips & Tricks

  • Use Specific Image Tags: Avoid using latest tag in production. Use specific version tags (e.g., python:3.9-slim-buster) for reproducibility.

  • Cache Layers: Docker caches image layers. Order your Dockerfile instructions to maximize cache reuse. Place frequently changing instructions (e.g., COPY . /app) towards the end.

  • Clean Up After Installation: Use && to chain commands in RUN instructions and clean up temporary files.

    RUN apt-get update && apt-get install -y --no-install-recommends some-package && \
        rm -rf /var/lib/apt/lists/*

  • Use a Linter: Use a Dockerfile linter like hadolint to identify potential issues in your Dockerfile.

  • Volumes for Development: Use Docker volumes to mount your source code into the container for development. This allows you to edit your code on your host machine and see the changes reflected immediately in the container.

    Terminal window
    docker run -it -v $(pwd):/app my-image:latest bash

  • Environment Variables: Pass environment variables to containers using the -e flag or in docker-compose.yml.

7. Integration

  • Pandas: Easily read and write data within Docker containers. Ensure the pandas library is installed in your requirements.txt.

  • Matplotlib: Generate plots and visualizations within Docker containers. You may need to use a headless backend like Agg to avoid requiring a display.

    import matplotlib
    matplotlib.use('Agg')  # Use a non-interactive backend
    import matplotlib.pyplot as plt
    

    # Your plotting code here
    plt.plot([1, 2, 3, 4])
    plt.savefig('plot.png')  # Save the plot to a file

  • Jupyter Notebooks: Run Jupyter Notebooks inside Docker containers. Map a port to access the notebook from your host machine.

    Terminal window
    docker run -it -p 8888:8888 -v $(pwd):/home/jovyan/work jupyter/datascience-notebook:latest start.sh jupyter lab --NotebookApp.token='' --NotebookApp.password=''
    • Important: This disables token-based authentication for simplicity. In production, use a more secure authentication method.

8. Further Resources

This cheatsheet provides a starting point for using Docker in your AI and Data Science projects. Remember to consult the official documentation for more detailed information and advanced features. Practice and experimentation are key to mastering Docker and using it effectively for creating reproducible environments.