Docker Model Runner, will it be a Ollama Killer ?

Docker Model Runner, will it be a Ollama Killer ?

---

---

AI Summary

Using Docker to Run Large Language Models Locally

Introduction

• Discusses running large language models (LLMs) locally using Docker.
• Introduces the new Docker Model Runner currently in beta.
• Collaboration with Michael Chiang, co-founder of AMA, who also worked on Docker Desktop.

Requirements

• Docker Desktop version 4.0 or higher is needed to utilize the features shown in the video.

Key Commands and Process

1. Pulling a Model: Use the command:

   docker model pull AI_small_M2  

   • This downloads the model.

2. Listing Models: To view all models on your machine:

   docker model list  

3. Inspecting a Model: To inspect details of a specific model:

   docker model inspect [model_name]  

4. Running the Model: Execute a model interactively:

   docker model run AI_small_M2  

   • Type /bu to exit the conversation with the model.

Model Hub

• Availability of a Model Hub featuring various models, similar to Docker Hub.

User Interface (UI) Version

• Demonstrates running a UI version via a shell script:

  ./run.sh  

  • Access the application at http://localhost:8081 for a chat interface.

Example Interaction

• Chat with the model, querying for information and requests for coding tasks.

Conclusion

• Highlights the potential of using local models via Docker, encouraging viewer feedback and exploration of LLM capabilities.