What happens when you send a problem report to Cakewalk?

New in SONAR X1 Expanded is SONAR’s built-in fault reporter. Aside from the fault reporter we have also have been using The Problem Report form on our website to collect feedback from users for many years now. Both of these tools do basically do the same thing – collect information so that our development teams can improve our products. In case you have ever wondered ‘what happens after I click the submit button’ or ‘does Cakewalk actually do anything with that information’ or if you just want a sneak peek about some of the new enhancements in SONAR X1d coming later this month then read on …

First a quick primer on debugging Windows software for non-geeks. Geeks feel free to skip ahead. Circa SONAR 8 (I actually don’t recall the exact version) a feature was added in SONAR that would automatically create a minidump file whenever something went wrong in SONAR. Whenever this does happen, aside from attempting to save a recovery version of your project file, SONAR also writes important information about the state of your computer, variables in your SONAR configuration (such as your sample rate and playback timing master) and more information that is extremely helpful in debugging into a minidump file. This file along with a clear set of steps taken that led up to the undesirable behavior gives a developer just about all the information they need to start debugging and fixing an issue.

OK, so you have this minidump file and filled out a problem report or you clicked the button in the fault reporter and sent your information to us. Here is what happens next:

  1. AutoBake3000 (I’d like to think of this as an evil robot of my creation, but really its just a small windows service that runs on a server in the Amazon Web Services cloud) wakes up about every 5 minutes and checks for new reports. AutoBake3000 is pretty smart and knows to look for both new reports via the Problem Reporter from our website and reports submitted with SONAR’s fault reporter.
  2. When you submit a report it also includes information about which version of SONAR you were using. AutoBake3000 uses this to launch and get the output from a Windows Debugger, specifically we use WinDbg.
  3. Finally, AutoBake3000 reads the output from WinDbg, compares it to all of the known issues we have information on and will either create a new case in our issue tracking system or append your information to an existing case. We’ll share more about our issue tracking system in another post.

So the entire above process takes about 30 minutes for most reports. It can actually be done in as quickly as 2 minutes in some cases but there are some moving parts involved in transferring uploaded files between servers. If the our website is getting a lot of traffic for example, the process that transfers minidump files to AutoBake3000 will wait until things have calmed down a bit.

So with AutoBake3000 we get A LOT of useful information, here are a few screen shots of the back-end reporting.

Minidumps processed by day
How many minidumps AutoBake3000 processed in the last 14 days
Reports grouped by function
Reports grouped by SONAR function from the last 14 days

…and yes that really tall column, that has been fixed in X1d.

Deciding what to focus on is very tricky, the SONAR triage team will prioritize issues that can cause crashes or data loss as the highest priority. The data from AutoBake3000 is incredibly helpful in that it helps the triage team get a good understanding of the scope of the issue and how many users are affected. Often times when a developer is working in one area of SONAR, it’s easier to fix other issues in that area as well. Here is a very small sampling of some improvements in X1d as a direct result of user feedback from the problem and fault reporters.

Resolved an issue where the Control Bar would appear blank when SONAR auto-saved in a minimized state.

If you have been using SONAR since X1 it is very likely you may have ran into this issue. While it’s not something that would cause data loss, it was incredibly frustrating. Unfortunately, it was incredibly hard to track down and resolve for good had it not have been for all of the helpful SONAR users providing feedback, details and taking the time to troubleshoot it. CWBRN’s 3962, 4381, 4317, 4072, 5323, 5489, 5770, 5788, 5882, 6302,  & 6870

Resolved a crash when inserting a soft synth and opening the Synth Rack if the Browser was not already opened

This is an issue that we suspected users we’re having troubles but it wasn’t until a problem report was submitted with helpful details that we were able to identify, isolate and resolve the issue. We only received one official report, but sometimes that is more than enough to fix an issue. CWBRN-6513

Various stability improvements

You have probably seen the line above in just about every update we have released. There was also recently a thread on the SONAR Forum asking about improvements to SONAR’s audio engine. The answer to the forum question is yes, with just about every release changes are made to improve performance. X1d is no exception to that with hundreds of changes based on user feedback and reports.  Often times there isn’t a clear title for an issue, sometimes they can be really hard to recreate. An example of how tricky debugging some issues can be from a recent email discussion about a particular ASIO-related crash issue that was fixed (CWBRN-5904)

Keith: “Turned out the use case was running a vendors ASIO panel outside of SONAR (via a Windows Control Panel applet) and then changing the sample rate or buffer size.  That sends an ASIO reset request, which if handled in our code a certain way at a certain time and the user had more than one ASIO device and the device in question wasn’t the first one (quite a few conditions), it could crash.”

Expect to see the complete list of updates and enhancements in February, when X1d is officially released.

Research Assistance/Partners in Crime:
Keith Albright – Principal Software Engineer/Evil genius behind SONAR’s built-in minidump creation
Ryan Munnis – Technical Support Manager/Problem fact-finding extraordinaire and Level 18 Minecraft Wizard