Implementing predictive analytical insights has become ever so essential for organizations to operate efficiently and remain relevant. What is important while doing this though is to be agile and adaptable. This is much so because what holds valid for a period can easily become obsolete with time. And what is characteristic of a specific group of customers, for example, varies widely with a diverse audience. Therefore, going from an envisioned innovative business idea to a working AI/ML model requires a mechanism that allows for a rapid and AI-driven approach.
In this post, I explain how Databricks, GitHub Copilot and Visual Studio Code IDE (VS Code) together offer an elevated experience when it comes to implementing predictive ML models efficiently. Even with minimal coding and data science experience, one can build, test and deploy predictive models. The synergy I’ve seen that GitHub Copilot has from within VS Code with MLflow and Databricks Experiments is remarkable. Here is how this approach goes.
Prerequisites
Before starting, there are a few one-time setup steps to configure VS Code so it’s well-connected to a Databricks instance. The aim here is to leverage Databricks compute (Serverless works too) which provides easy access to various Unity Catalog components (such as tables, files, and ML models).
- In VS Code, connect to GitHub Copilot
- Install the Databricks Extension for VSCode
- Configure a Databricks project in VS Code
Define the Predictive Modeling Agent Prompt in Natural Language
Use the GitHub Copilot Agent with an elaborate plain language prompt that provides the information it needs to devise the complete solution. Here is where the actual effort really is. I will list important points to include in the agent prompt that I discovered produce a more successful outcome with less iterations.
- Data Sources: Tell the Agent about the source data, and not just in technical terms but also functionally so it considers the business domain that it applies to. You can provide the table names where it will source data from in the Unity Catalog and Schema. It also helps to explain the main columns in the source tables and what the significance of each column is. This enables the agent to make more informed decisions on how to use the source data and whether it will need to transform it. The explanations also result in better feature engineering decisions to feed into the ML models.
- Explain the Intended Outcome: Here is where one puts their innovative idea in words. What is the business outcome? What type of prediction are you looking for? Are there multiple insights that need to be determined? Are there certain features of the historical data that need to be given greater weight when determining the next best action or a probability of an event occurring? In addition to predicting events, are you interested in knowing the expected timeline for an event to occur?
- Databricks Artifact Organization: If you’re looking to stick to standards followed in managing Databricks content, you can provide additional directions as part of the prompt. For instance, what are the exact names to use for notebooks, tables, models, etc. It also helps to be explicit in how VS Code will run the code. Instructing it to use Databricks Connect using a Default Serverless compute configuration eliminates the need to manually setup a Dataricks connection through code. In addition, instructing the agent to leverage the Databricks Experiment capability to enable model accessibility through the Databricks UI ensures that one can easily monitor model progress and metrics.
- ML Model Types to Consider: Experiments in Databricks are a great way of effectively comparing several algorithms simultaneously (e.g., Random Forest, XGBoost, Logistic Regression, etc.). If you have a good idea of what type of ML algorithms are applicable for your use case, you can include one or more of these in the prompt so the generated experiment is more tailored. Alternatively, let the agent recommend several ML models that are most suitable for the use case.
- Operationalizing the Models: In the same prompt one can provide instructions on choosing the most accurate model, registering it in a unity catalog and applying it to new batch or stream data inferences. You can also be specific on which activities will be organized together as combined vs separate notebooks for ease of scheduling and maintenance.
- Synthetic Data Generation: Sometimes data is not readily available to experiment with but one has a good idea of what it will look like. Here is where Copilot and python faker library are advantageous in synthesizing mockup data that mimic real data. This may be necessary not just for creating experiments but for testing models as well. Including instructions in the prompt for what type of synthetic data to generate allows Copilot to integrate cells in the notebook for that purpose.
With all the necessary details included in the prompt, Copilot is able to interpret the intent and generate a structured Python notebook with organized cells to handle:
- Data Sourcing and Preprocessing
- Feature Engineering
- ML Experiment Setup
- Model Training and Evaluation
- Model Registration and Deployment
All of this is orchestrated from your local VS Code environment, but executed on Databricks compute, ensuring scalability and access to enterprise-grade resources.
The Benefits
Following are key benefits to this approach:
- Minimal Coding Required: This applies not just for the initial model tuning and deployment but for improvement iterations also. If there is a need to tweak the model, just follow up with the Copilot Agent in VS Code to adjust the original Databricks notebooks, retest and deploy them.
- Enhanced Productivity: By leveraging the Databricks Experiment APIs we’re able to automate tasks like creating experiments, logging parameters, metrics, and artifacts within training scripts, and integrate MLflow tracking into CI/CD pipelines. This allows for seamless, repeatable workflows without manual intervention. Programmatically registering, updating, and managing model versions in the MLflow Model Registry, is more streamlined through the APIs used in VS Code.
- Leverage User Friendly UI Features in Databricks Experiments: Even though the ML approach described here is ultimately driven by code that is auto generated, that doesn’t mean we’re unable to take advantage of the rich Databricks Experiments UI. As the code executes in VS Code on Databricks compute, we’re able to login to the Dababricks interactive environment to inspect individual runs, review logged parameters, metrics, and artifacts, and compare different runs side-by-side to debug models or understand experimental results.
In summary, the synergy between GitHub Copilot, VS Code, and Databricks empowers users to go from idea to deployed ML models in hours, not weeks. By combining the intuitive coding assistance of GitHub Copilot with the robust infrastructure of Databricks and the flexibility of VSCode, predictive modeling becomes accessible and scalable.
Source: Read MoreÂ
