Dentrix services not starting on boot (2023 edition)

Last week, we experienced catastrophic failures with Dentrix. Every attempt to launch Office Manager, Appointment Book, Family File and Ledger, on the server and all workstations, crashed within seconds of launch.

Our system:

  • Dentrix 23.2.3.6311
  • Windows Server 2019 Essentials
  • 15+ workstations

We were without Dentrix for 6 hours while Dentrix Support uninstalled then reinstalled (!) Dentrix. That resolved it at the time, but it was not clear as to why it worked (and Support had no idea either).

A week later, Windows Update (KB5026362) downloaded on a Thursday, but did not yet install.

We restarted the server on the following Saturday and let the Windows Update complete. On Sunday, we attempted to launch Dentrix. Appointment Book and Office Manager would spin for a few seconds, then crash and shut down, on the server and on all workstations. Mild panic ensued. Interestingly, Document Center, Office Journal, and Timeclock still worked.

We attempted these things, none of which worked:

  • We re-enabled LLMNR
  • We uninstalled KB5026362

Lots of Google searching and deep thinking, we looked at Server Manager, and noticed that not all services were being launched at start (there was a red 5 next to it, indicating that 5 services did not launch). One of those was DtxUpdaterSrv. It listed status of “Stopped” but startup type of “Automatic”. Which means, it attempted to start but couldn’t. On a whim, I started the service manually, and immediately the broken applications started working

Digging deeper, I saw that these services, along with DentrixACEServer, all had attempted to start on system boot. None of them completed; every one of them had an Event log that showed “A timeout was reached (30000 milliseconds) while waiting for the … Service service to connect”.

With some searching, I came across this ServerFault post on how to change the default timeout from 30 seconds to 60 seconds. I applied the change and rebooted. The Dentrix services came up on the reboot, so things may be better! Only time will tell of course.

Escaping the present

From an external news perspective, 2017 has been a difficult year. I find the current administration to be full of dangerous nitwits who will either get us all killed or leave our way of life severely weakened.*
I found myself continually sad at the end of each night during the first quarter. I would read the news or browse Facebook (which consisted mostly of news for me) right before going to bed, then be depressed at what the next four years would bring.
Come Q2, and specifically a long weekend trip we planned to take in early June, I felt an urge to read fantasy/sci-fi. I recalled some elements of Brandon Sanderson’s takeover of The Wheel Of Time series, but felt like the last three books were worth another read. And so, I picked up each of books 13, 14, and 15, and read them at nights, during free time, and more.
I didn’t stop there. After plowing through them in 2 months, I then decided to re-read Sanderson’s The Stormlight Archive series. I finished those two books in another month.
And looking back, even though there were moments where I was definitely overdoing it (somewhat ignoring my family, sneaking pages in here and there), I was way, way happier falling asleep than when I read the news or Facebook.
So, here’s to hoping I can find a happy medium!
*Every other part of my life has been wonderful. I guess I can’t complain too much.

Getting a Dell OptiPlex back up and running with Microsoft Windows 7

This post chronicles the month-long adventure in getting one of our Dell OptiPlex 3020M (you know, the tiny one with no CD/DVD drive) desktops back in good working order.

First, the problem:

  • Boot times start taking extremely long to get to the Windows loading screen, and then it would spin endlessly
  • Safe mode boot didn’t help this process along

As we were still under warranty with Dell, I reach out to them. Here is what happened:

  • I emailed Dell Support. They replied very quickly, and after a couple of days of back/forth with their tools, we isolated a bad hard drive.
  • I went through Dell Diagnostics with them (which is roughly baked into the BIOS). Turns out the hard drive was bad.
  • They sent me a replacement hard drive via FedEx Overnight.
  • I swapped it out and started the desktop. After booting to Windows and getting through a few prompts, I get stuck in a never-ending Dell configuration loop that is (what I call) the gray-on-black modal dialog of hell. Loop runs for 24 hours with no progress or hard drive activity. I can’t do anything (mouse and keyboard are disabled).

Dell OptiPlex 3020M stuck in gray on black loop of hell

Dell OptiPlex 3020M stuck in gray on black loop of hell

  • Dell Support directed me to download an .iso file that contains a preconfigured Windows 7 for my system. I didn’t know such a thing exists! Now I want to go get one for every one of the machines we maintain…
  • I download it, burn the .iso to USB with Rufus on the Surface Pro we have here (remember, no DVD ROM drive on the desktop I’m trying to recover), and try to get it to run.
  • I am able to boot to it and get through a few prompts. But very soon, I encounter a new error: “A required CD/DVD drive device driver is missing.”

"A required CD DVD driver is missing" dialog

“A required CD DVD driver is missing” dialog

  • I tried a number of things suggested by others on the Internet, including: switching the USB to another port midway through the installation, downloading Dell’s drivers and having Windows attempt to find the correct driver, buying an external CD/DVD writer, trying to burn the .iso to DVD (doesn’t work), and trying to just install Windows 10 from a later Dell machine onto this machine (also doesn’t work). Many hours were wasted.
  • I reach back out to Dell and complain about how much time I’ve wasted. After complaining, Dell Support sent someone onsite to replace the hard drive (again). After this was done, I was back at the neverending gray-on-black loop of hell.
  • I opened a new support ticket with Dell, this time around OS issues. I was sent new instructions for creating the Windows 7 Recovery USB key, which largely maps to this article. The article was very interesting, because:
    • It recommended using the diskpart Windows command line tool, vs. Rufus which I used previously, and
    • It didn’t quite work as is, as there was one significant discrepancy I discovered from those directions. I have a 64GB USB 3.0 key. I had to create a 16GB FAT32 partition (which is known to boot well; I can’t trust NTFS or ExFAT as boot partitions) vs. simply creating a full-disk 64GB partition. It took me a few tries to figure that out.
  • Success! This got me through the two key issues I had in the past (gray-on-black and missing device driver)
  • But, I encountered another problem: an error that said “Windows cannot be installed on this disk. The selected disk has an MBR partition table. On EFI systems, Windows can only installed on GPT disks.” Thankfully, this link indicated that I can safely delete all of the existing HDD partitions and create a new, big one. I did so, and the installation continues.

I wish the process wasn’t as difficult as it was. But at least I’m thankful that it works, and that my interactions with Dell Support helped me down the correct path, eventually. They were patient during the process, and had no problem sending their onsite tech to try to assist, which is the best I can hope for with business-level support.

A dead simple way to improve your writing

Why does anyone think using the phrase “dead simple” is remotely acceptable?

Example:

DishwasherNow is the dead simple way to have a brand new dishwasher delivered to your door from iPhone.

Is “dead” an adjective for “simple”, which has morphed into a noun but has also ceased functioning in this world?

Or is this statement implying that the service is so easy to use that the walking dead can even order their dishwasher from the iPhone they are carrying in their lifeless hands?

Why can’t we just use “easy” or “simple”?

DishwasherNow is the easy way to have a brand new dishwasher delivered to your door.

See how much better that reads? You don’t have to be a rockstar engineer or ninja growth hacker to understand what this phrase means.

EDIT: I’m on the losing end of this battle (from Google Books Ngram Viewer):

Dead Simple usage in Google Ngram Viewer

A practiced method to solve hard problems

Here at Mediafly, we are faced with hard engineering, product, sales and marketing problems every day. Each of us takes a different approach to solving these problems. Some of us like to create pros/cons lists. Others dig deep into data and use that help answer every question. No one approach is the “right” approach for everyone.

I recently had a conversation with our Engineering Manager[1], and he described his approach to solving hard problems.

  • Take an attempt to solve the problem, but don’t stress about it if you can’t figure out the solution yet.
  • Review the key aspects of the problem right before you go to sleep. This involves working on the problem from multiple angles. Meaning, if you tried one solution and it doesn’t work, try another. If it did work but it’s ugly, just note the key parts of why it works and why it’s ugly. You need to get intimate with the problem and be really familiar with it from all angles.
  • Now, let the problem go. Sleep on it, take a shower, go for a run. Do something to take your mind off of it entirely.
  • When you least expect it, an insight will find you. When the solution does find you, immediately explain it to as many people as you can. Don’t worry about whether they are an expert in the subject domain. Just start explaining. The mere process of explaining acts as a forcing mechanism to refine the solution further. It also serves as a filter; if what you thought was initially a great idea turns out to not be, attempting to explain may allow you to filter out the seemingly-good idea much more quickly, and get back to solving the problem another way.

I’ve watched him apply this method of problem solving over the years, and it truly is a thing of beauty. He will often take 2, 3, 4 attempts at particularly thorny engineering problems. He will sometimes throw away the code he wrote for an attempt and go back to the drawing board. He will restart this process from scratch as necessary. But, regardless, he almost always comes up with a solution that solves the problem elegantly. And watching his success has led me to begin adopting this approach for problems of all sort that I face as well.

[1] Special thanks to @laimis for being the inspiration for this method and this post! And he credits A Technique for Producing Ideas as inspiration for this process.


Dear Enterprise Software Product Managers: Consider Scale

5425752305_7023df6048_o-1

Discussions about design and UX for software these days so often focus on onboarding. Scott Belsky, founder of Behance, even suggests A good discipline to help you stay simple is to focus at least 50% of your effort on onboarding and the first-time-user-experience. Providing a great onboarding flow is the quickest way for your users to find value in your new feature. After all, the sooner a new user is able to find value, the more sticky it’ll be for them, and the less churn you’ll experience, right?

Makes sense, and it gives a great starting point for how to think about a new feature. For example, from Mediafly’s point of view:

  • A newly signed-up user will start with 0 content, 0 salespeople, 0 users. Envisioning that scenario is very straightforward.
  • A small business might be using SalesKit by Mediafly to manage 50 pieces of sales collateral among 20 salespeople, distributed to 500 prospects. There is some complexity with this level of information, but for most features you might design, it’s probably pretty straightforward.

Often, however, this is where the design of a feature stops.

When you’re working with enterprise organizations with large user counts, diverse business processes, very large data sets, or whatever key metrics you track, however, you need to consider the user experience when there is high volume of use in these key metrics.

Example 1.

From the beginning, we designed Mediafly’s content management system (CMS), Airship, to start as simply as possible. From day 1, users could drag in content from their laptop, with reasonable defaults, and immediately get value. As our customers adopted our CMS and scaled out their use across diverse business processes and groups, we continued to discover issues that we could never have foreseen at launch.

Recently, a large customer (a major CPG enterprise)began uploading merchandising layout diagrams, hierarchically organized by region, for each of their tens of thousands of their customers’ stores to our system. This dramatically increased two key metrics: their volume of content (tens of thousands of new documents) and frequency of updates (thousands of changes every week).

Automating the upload and management process on their end is a no-go, as there is no common backend system where these documents reside. And, asking people to update these layout diagrams with our Airship CMS would require 20-40 hours a week of navigating, clicking, and dragging/dropping.

To address this, we conceived of a new upload model in which an adminstrator of a one of their region’s merchandising layout diagrams could organize the new content hierarchically on their laptop, zip up the file, and upload it into our system. We would then interpret the results and update the content automatically in the correct location. This solution at once solves both the problem of content volume and update frequency. And it can be reused for other customers who encounter similar challenges.

We spend as much time solving user experience challenges of scale as we do thinking about how to build compelling new features whose adoption will begin at very low volumes.

Example 2.

We recently released the ability for our content administrators to create special links to view content, which has been a hit with our Media/Entertainment customers. The link can have a password, be tied to a user account, or be public. Creating a link is straightforward, and initial reception and usage of this feature started off as very positive.

But, after a few months, we began to hear feedback from content admins about some challenges they were feeling as their use cases for links expanded. The volume increased dramatically in some of these use cases. We now see that some admins have to create as many as 200 individual links for individual users in a single day, usually around television pilots or key screening seasons.

After diving deeper into some of these workflows, we created a process diagram to show what the typical process is to create a link. The content admin:

  • Switches to their email client and composes a new email
  • Pastes in a template that they use for their emails
  • Switches to Airship
  • Finds content in the hierarchy
  • Navigates to the Links tab
  • Taps Create Link
  • Configures the link
  • Saves the link
  • Clicks the option to copy the link to the clipboard
  • Switches back to their email client
  • Pastes the link in
  • Sends the email

Whoa, that’s a lot of steps. Imagine having to go through this process 200 times in one day! For some admins, it requires the entire day.

We have since simplified the process to create a large number of links, and continue to improve upon the feature to solve the problem of high volume even further.

How these experiences have changed us

As we design new features, we now include an extra question to answer: What will this look like at high volume? At the design phase, we strive to have a hypothesis on how we would address problems of scale, and to see what we can do to simplify the initial UX even further should scale arrive faster than we can roll out a redesign.

However, just like most things we do from a product and engineering perspective, we operate iteratively. We certainly won’t prematurely optimize for scale. But by simply adding this question to our checklist of considerations, we’ve opened up the ability to solve the seemingly inevitable high volume issues that will arise.

(This post was cross-posted from the Mediafly Blog.)

Announcing SalesKit Meeting Tracker

(Cross-post from Mediafly and LinkedIn)

Here at Mediafly, we work hard to make the lives of salespeople better. We talk to a lot of salespeople, and what we hear from reps over and over again is “I hate my CRM.” This feeling comes from many reasons:

  • While CRM is very valuable to sales management, salespeople find these systems add little to no value to their primary job as salespeople
  • After a long day or week of meetings, salespeople have to drag themselves into their CRM to record notes of what happened in each sales meeting they had—instead, they want to be working towards their next deal or spending time with their families
  • CRM systems in general are clunky, complex, and require a lot of clicks/taps to accomplish the most basic of tasks

What does SalesKit Meeting Tracker do?
Once SalesKit Meeting Tracker has been turned on for your company’s environment, tap on the “Meetings” button and start tracking your meeting.

Start Meeting

When you’re done, stop tracking, record the meeting’s details, and send them to Salesforce.com.

 

Complete Meeting

Your meeting’s details will appear within the Activity History for the account, contact, and opportunity you’ve chosen.

What’s coming next?
We have a rich roadmap for SalesKit Meeting Tracker, including integration to our other app platforms (web, Windows, Mac, Android) and to other CRM platforms (Microsoft Dynamics, SAP Cloud for Customer).

Navigating the Product Roadmap

Navigating the Product Roadmap

As is the case at many other startup/early growth phase companies, we at Mediafly haven’t hired a product manager. “Product management may be the one job that the organization would get along fine without”, and we’ve lived by that model for the past few years. Because of this, we operate lean and create our own best practices for the meaty parts of product management, most notably the product roadmap.

The product roadmap is necessary for any enterprise-focused SaaS company that is selling multi-year subscriptions to Global 2000 companies like we are. It’s used as a communication tool for:

  • Clients, both the ones signing the checks as well as the heaviest users/admins of your product. They want to understand (and influence) what is coming down the pipeline for your product
  • Product development team members, including designers, developers, QA, and infrastructure. They want to understand what they should be thinking about from a UI/UX/architecture perspective
  • Sales and marketing. They want to know what they can sell, how it should be marketed, and provide input from a client, competitive and market-driven standpoint

Defining the Roadmap

An Example of Mediafly’s Product Roadmap

It can be easy to let the roadmap balloon into a complex, time-sucking monster. The roadmap itself is housed in Google Sheets, and broken up by product category (for us, those are SalesKit, ProReview, Airship CMS, Interactives, Reporting, andInfrastructure/Security). We organize specific entries in the roadmap into five columns, with the following definitions:

  • Discovery: Ideas/features that are “baking.” We usually hear of these ideas from clients, prospects, users, or the market, and are looking to gather more information for these ideas before we move them to the next phase.
  • Evaluation: Once an idea has gathered sufficient input, it moves into the Evaluation phase. During this phase, we’re putting together benefits statements, wireframes, high-order engineering estimates, and public-facing selling documentation.
  • Plan: After we’ve committed to building an idea as a feature, we attach a delivery date (usually a quarter, sometimes a month), and move it to Plan. Here, we are working on detailed requirements, wireframes, and phasing. We are also conducting usability tests and starting to talk about the upcoming feature with our clients and prospects.
  • Build: Once the feature reaches our design and engineering teams, it is in Build.*
  • Release: Recently released features are moved here. At this point, we’re training our clients, updating documentation, and analyzing usage in preparation for improving our newly released feature.*

It’s All in the Details

Sample Mediafly Details Document

Some entries in the product roadmap will be hyperlinked to a Details Document, which is hosted in Google Docs alongside the roadmap itself. These documents will vary in structure, but often contain:

  1. Estimates
  2. Requirements
  3. Current and proposed approaches
  4. Discussion of tradeoffs and limitations
  5. Specific customer requests

Every 3-4 weeks, I will review the roadmap with our business leaders (currently, our CEO and EVP of Sales and Marketing). We focus on what’s changed since our last meeting and discuss new data that we’re seeing from clients, prospects, and the market. Then every 2 weeks, I will review the roadmap with the Product, Engineering and Customer Success teams. We focus on what’s coming up in the near future, and often will dive into the specific points to discuss better ways to solve specific problems.

Ad-hoc, I will be asked to present the roadmap to our customers. This is usually accompanied by a short presentation that explains what the roadmap is and how it works. Over the course of any week, I’ll spend about 1-2 hours reviewing and adjusting entries within the roadmap based on conversations, research, and incoming data.

We’ve managed to keep the product roadmap useful and relevant without too much overhead over the past few years by following this model. As we grow the entire Mediafly team through 2016 and beyond, this process will certainly evolve. I’m looking forward to that evolution, and will keep you up to date as it progresses.

If you’re involved in product management at your organization, please comment below with your best practices. I’d love to hear from you!

(Cross-posted on the Mfly Blog)