A Day in the Life of a Machine Learning Engineer
What does it actually take to build AI that performs reliably in the real world? We sat down with Shilpa Ullas, Machine Learning Engineer at DeepAlert, to talk about life beyond the benchmark — from investigating unusual edge cases and tuning real-world deployments to those rare “heroic detections” that make the whole team take a second look.
For a machine learning engineer working in video analytics, building the model is only part of the job.
The real challenge begins when that model encounters thousands of different cameras, environments, lighting conditions and real-world scenarios — many of which look nothing like the neatly controlled datasets used during development.
Walk us through a typical day. How much is model development versus digging into deployment data?
One of the things I enjoy most about my role is that no two days are the same.
A typical day is a balance between improving machine learning models and understanding how they perform in real-world environments. That could mean investigating customer-reported cases, analysing data to uncover the root cause of an issue, evaluating model improvements, or working closely with the team to ensure models are deployed reliably in production.
The role goes far beyond simply building models. It’s about solving real-world problems and continuously improving AI systems to deliver meaningful impact.
What tools do you spend most of your time in?
Most of my work revolves around Python, where I analyse data, evaluate model performance and investigate challenging edge cases.
I also rely heavily on visualisation tools to inspect images and model predictions, because they often reveal insights that metrics alone cannot.
For experiment tracking and comparing different model versions, I use platforms such as MLflow to keep results organised and reproducible.
Together, these tools help bridge the gap between model development and understanding how AI actually performs in real-world scenarios.
AI surfaces the event. People make the decision.
One of the recurring themes in machine learning is that the real world has an impressive ability to produce situations nobody anticipated.
Across the deployments you have visibility into, what’s a failure mode customers might not expect?
The biggest surprise is that there is always another edge case.
Just when you think you’ve seen everything, a new environment or scenario challenges the model in an unexpected way.
I don’t necessarily see those as failures. They’re opportunities to better understand how the model behaves outside controlled datasets. Every deployment expands our understanding of the real world and helps us build more robust AI systems.
Are there particular environments that consistently make detection more difficult?
Real-world environments are inherently dynamic, and that’s what makes computer vision both challenging and exciting.
Factors such as lighting, camera placement, object distance and scene complexity can all influence how a model performs.
Rather than optimising for one perfect scenario, the goal is to build models that generalise well across a wide range of conditions.
Not every surprise is a difficult edge case. Sometimes the model does something that exceeds even the engineering team’s expectations.
What’s the strangest or most impressive thing the model has correctly detected?
One of the most rewarding moments is when the model delivers what we informally call a “heroic detection.”
There have been quite a few occasions where it has successfully detected a target under extremely challenging conditions — objects that are barely visible because of poor lighting, very low contrast or significant distance from the camera.
Those moments genuinely stand out because they often exceed our expectations.
They’re the kind of detections that make you pause and take a second look.
How do those stories usually reach you?
They usually surface during our regular reviews.
We spend a significant amount of time analysing deployment data to understand how the model behaves across different environments.
While much of that work focuses on identifying areas for improvement, it’s equally rewarding to come across exceptional detections that demonstrate the model’s capabilities under challenging real-world conditions.
In video analytics, more alerts don’t automatically mean better performance. An effective system needs to surface events that are useful within the customer’s actual operational environment.
When a customer reports that a site is too noisy, what happens on your end?
The first step is understanding what “noisy” means for that particular deployment, because it can differ significantly from one site to another.
We begin by reviewing the reported events and looking for recurring patterns: the time of occurrence, environmental conditions, camera placement or particular characteristics of the scene.
The goal is to determine whether the alerts represent genuine detections or whether site-specific factors are reducing their operational value.
Every deployment has its own environment and requirements, so understanding that context is essential before deciding what needs to change.
Is tuning mostly threshold adjustments, configuration changes or retraining?
Effective tuning is about understanding the root cause rather than applying a standard fix.
Some situations can be addressed through deployment-specific configuration, while others reveal opportunities to improve the underlying model.
The key is determining whether an issue is isolated to a particular environment or represents a broader pattern across multiple deployments.
That approach doesn’t just help optimise an individual site. It can also drive improvements that benefit the wider customer base.
How do you know when a site is properly tuned?
A site is well tuned when its performance aligns with the customer’s operational needs.
That requires an ongoing feedback loop: reviewing performance, understanding customer observations, implementing improvements where necessary and validating the outcome in real-world conditions.
Model metrics provide valuable insight, but they don’t tell the whole story.
Success ultimately comes from delivering a solution that performs reliably in production and gives customers confidence in the alerts they receive.
DeepAlert builds the video analytics software used by security companies and monitoring teams. It doesn’t replace the people responsible for deciding how an event should be handled.
How do you think about the line between what DeepAlert’s models do and what the customer’s monitoring team does?
We see AI as a decision-support tool rather than a decision-maker.
Our responsibility is to build models that accurately identify and surface relevant events. The customer’s monitoring team then applies its expertise, operational knowledge and established processes to determine the appropriate response.
That partnership between AI and human judgement is what enables effective video analytics: technology enhances situational awareness while people remain responsible for making the critical decisions.
What do people most often misunderstand about what an ML engineer at a video analytics company actually does?
One common misconception is that building an accurate model is the finish line.
In reality, that’s just the beginning.
The real challenge is developing AI systems that perform consistently across diverse real-world environments.
Success isn’t defined by benchmark scores alone. It’s measured by how reliably the solution delivers value in production.
Machine learning is often portrayed as a highly individual technical discipline. In practice, developing production AI is very much a team effort.
Anyone you’d like to give a shout-out to for helping solve a problem or improve a model recently?
Everyone on the team brings something unique to the table, so it’s difficult to single out just a few people. That said, based on my own collaborations, I’d like to give a special shout-out to Arun Aniyan and Bradley Culligan.
Arun has an exceptional ability to bring clarity to complex problems, and I genuinely admire the vision and perspective he brings through his experience.
Bradley has been a fantastic sounding board for discussing new ML approaches. His enthusiasm and ideas have often challenged my thinking and led to valuable insights.
I’m grateful to work alongside teammates who are always willing to share their knowledge and learn from one another.
1. Being an ML engineer can feel a lot like detective work
Every edge case tells a story.
Finding out why something happened — rather than simply observing that it happened — is often where the biggest improvements begin.
2. The strongest AI systems still need people
AI is sometimes framed as a replacement for human involvement.
In practice, some of the best outcomes come from people and AI working together. AI can rapidly identify patterns and surface important events, while human judgement and context help determine what those events mean and what should happen next.
3. Curiosity might be the most important skill
Curiosity is one of the most valuable qualities you can have when working in AI.
Every unexpected result is an opportunity to learn something new — and those lessons often become the foundation for the next improvement.
The work of a machine learning engineer doesn’t end when a model reaches a certain accuracy score.
For Shilpa and the DeepAlert ML team, the real test happens when that model leaves the development environment and encounters the unpredictability of the real world.
Every unusual camera angle, difficult lighting condition, noisy site and heroic detection adds another piece to the puzzle.
And that continuous cycle — observe, investigate, learn, improve — is how production AI gets better.
Related Articles