This course covers Generative AI engineering on the platform Azure Databricks, using Apache Spark to explore, fine-tune, evaluate, and integrate advanced language models. Participants will learn to implement techniques such as Retrieval-Augmented Generation (RAG) and multi-stage reasoning, tune language models for specific tasks, and evaluate their performance. The course also emphasizes the principles of Responsible AI and managing models in production through LLMOps (Large Language Model Operations) in Azure Databricks.
The course is recommended for:
- Data Scientists who develop and evaluate Generative AI applications.
- AI Engineers who implement language models at scale in Azure Databricks.
- Machine learning and artificial intelligence professionals familiar with basic AI concepts and the platform Azure Databricks.
After completing the course, you will know how to:
- Explore and use Large Language Models (LLMs) in Azure Databricks.
- Implement Retrieval-Augmented Generation (RAG) techniques for more accurate and contextual results.
- You build multi-stage reasoning systems to solve complex problems.
- You fine-tune linguistic models and specialize them for specific tasks.
- Evaluate the performance of LLMs using LLM-as-a-judge metrics, techniques, and methods.
Participants should have:
- Fundamental knowledge of artificial intelligence and machine learning.
- Familiarity with the basic concepts of Azure Databricks.
- Introduction to Large Language Models (LLMs) in Azure Databricks
- Basic notions about LLMs and their applications (text summarization, sentiment analysis, translation, etc.)
- Creating and using interactive reports with LLMs
- Implementation of Retrieval-Augmented Generation (RAG)
- Integrating search engines with generative models
- Creating more accurate and contextually relevant outputs
- Implementing multi-stage reasoning
- Approaching complex problems through sequential steps
- Integrating partial results into a complete reasoning process
- Fine-tuning for Large Language Models
- Adapting LLMs for specific tasks
- Reducing costs and improving model relevance
- Evaluating linguistic models
- Metrics and methods for evaluating LLMs
- Challenges and good practices in evaluation
- Automated techniques, including LLM-as-a-judge
- Responsible AI for language models
- Principles of responsible implementation
- Ethical considerations and risk reduction
- Using security tools for LLMs
- Implementing LLMOps in Azure Databricks
- Basics of LLMOps
- Monitoring, managing and maintaining LLMs in production
- Microsoft AI-3026: Develop AI Agents on Azure
- Microsoft DP-3011: Implementing a Data Analytics Solution with Azure Databricks
- Microsoft DP-500: Designing and Implementing Enterprise-Scale Analytics Solutions Using Microsoft Azure and Microsoft report
There are no certification programs at this time.
