How Toluna Ensured Full Visibility During Cloud Migration to AWS

Before adopting Coralogix, the DevOps team at Toluna was investing about 10% of their time maintaining an open source log management solution.

Based on how many logs were expected, the team would split the data across 10 indices with an added buffer to avoid data loss. To streamline their work, the team created automation to delete old indices and manage them according to different priorities and criteria. Still, they had to be on-call to ensure the reliability of the system, especially in the case that an error would cause a huge spike in logs and the indices would become full.

When that happened, the team would often be woken up in the middle of the night to delete data and get the system working again. Everyone would complain about the effectiveness of the system to the point that it affected the team’s reputation in the company. And, of course, they don’t have control over what is being written to the logs and unexpected data can come through and cause issues at any time.

The DevOps team was spending an unreasonable amount of time and resources just to survive, just to make sure the systems were up and running. At the same time, the team was frequently needing to optimize the clusters to reduce latency and poor performance.

With Coralogix the DevOps team has direct control over how the data is processed from within the Coralogix platform. The team can decide not to index that data and instead move it to only be stored in their archive bucket where the team will still be able to explore it so there’s never any coverage loss.

In the case that the data is producing a lot of noise, it can easily be blocked. Both of these options allow the team to basically ingest a lot more data without worrying about any performance or cost issues.

Adopting Coralogix has shifted the nature of the work that DevOps is focused on. Instead of spending 10% of their time managing their in-house solution, the team is able to work on optimizing and making significant improvements to the workflows across the business and R&D organization.

The implementation was simple, with all AWS services naturally integrating with the Coralogix platform. During the onboarding process, Coralogix’s customer success team worked with Toluna to set up custom parsing and value extraction to improve their performance and reduce storage requirements.

Having this centralized platform where developers can access all of the logs is a game changer on its own for the team’s productivity. With the previous solution, log data from each index would need to be accessed separately, and permissions would need to be provided to each instance as well.

Now, it’s really easy to authenticate new users and they can access the data they need from a single screen. It’s much more comfortable now for them to work and understand, at a glance, if and why there’s a problem.

Especially during a cloud migration, system visibility is critical. Using Coralogix, the team was able to easily compare different versions, see changes in system behavior, and proactively identify issues using anomaly detection capabilities.

“We’re able to see things we never saw before, draw new conclusions about our systems, and just make sense of what’s going on in our production.”

The Loggregation feature is a powerful tool being used by Toluna to quickly investigate and resolve issues. The DevOps team is now using the time they’re saving on managing their own solution to develop new use cases for the organization’s observability data, but even with the initial functionality of monitoring and investigating errors, they see a huge amount of value.

Things that would take days now take between hours to minutes.

Despite the organizational preference to build and manage tooling in-house, the team saw that Coralogix could save them thousands of dollars a month. Coralogix further improves Toluna’s ROI on their observability investment by freeing up their DevOps team, improving developer productivity, and decreasing the time to identify and resolve production issues.

Adoption in the organization is high, with more than 100 users from across the R&D organization. Everyone is using it – from developers, DevOps and QA to product managers, Support and even some business roles.