WinSxS, SysWOW64, and Installer too large

When attempting to upgrade Dentrix on our relatively new server, we encountered a “Not enough space left on C:” drive. What? We have barely anything installed on this server.

For reference, this server has a single physical 1TB hard drive. That drive is logically partitioned into C: (60GB) and D: (the rest). D: is where we place all Dentrix, images, etc. data. The server is relatively new (2022) and doesn’t have the years of accumulated cruft that older servers have.

I ran WinDirStat to get a visual of where the hard drive was being used:

Odd.

Some initial searching on Google revealed this Microsoft reference page to clean up WinSxS.

First, in an admin PowerShell, I ran:
dism /Online /Cleanup-Image /AnalyzeComponentStore

This returns:

  • Actual size on disk
  • Size of reclaimable data
  • Whether a cleanup is recommended

The answer to the third question was “yes”, so we ran the cleanup with:
dism /Online /Cleanup-Image /StartComponentCleanup

This only got us back to 8.5GB free (out of 60GB).

I followed up with:

dism.exe /online /Cleanup-Image /StartComponentCleanup /ResetBase

We are now up to 9.5GB free. Windows is now consuming 36.3GB of the C: drive.

In theory, this wasn’t enough to initiate the Dentrix upgrade. The Dentrix installer claims it requires 20GB (!) of free space on C: to continue. But, we tried the Dentrix upgrade again, and it went through (!!).

I’m out of options when it comes to clearing out more space on C: from these bloated Windows files. Everything I read online indicates that this is just the way Windows is, and there isn’t more we can do.

So, I feel like the next best option is to repartition D: and C: to give C: more of D:’s empty space.

Of course, reparitioning is Scary and Dangerous. I need to do a full disk backup before I do that. Which means pulling out Rufus and probably buying a giant external hard drive, before I do this.

What a mess.

Introducing Vocab Word Study

I built an app called Vocab Word Study. It’s on the iOS App Store and it’s currently free.

I built it to help my 6th grade kids start to build a stronger vocabulary[1]. I didn’t like what was out there, and wanted something that: 

  1. Is very simple and short to use on a daily basis, 
  2. Requires them to write a sentence using the word, and 
  3. Is very specific to the kinds of words they are going to see in literature and on standardized tests in high school and beyond.

Try it out, and give it a 5-star rating if you can!

How we speed up Dentrix clients on our network

For our various office networks with Dentrix clients and Dentrix server (we are running standard Dentrix, not Enterprise edition and not Ascend), here are a list of things that are part of our provisioning checklist that we do to speed them up:

Disable LLMNR

LLMNR (Link-Local Multicast Name Resolution) is a protocol based on the Domain Name System packet format that allows both IPv4 and IPv6 hosts to perform name resolution for hosts on the same local link. It allows host name resolution without the requirement of a DNS server. While it’s in theory useful, in practice we have not needed on our networks for at least a decade now. And in certain scenarios disabling it both speeds up our network and slightly improves security in the case of an incident. Here is how to disable LLMNR, and some research that we’ve discovered.

Programming note: If you need to convert Dentrix PDF reports to Excel, Google Sheets, or CSV, try Dentrix Report Convert. This is a simple service I’ve built to automate this. It is excellent for practice owners, office managers, accountants, attorneys, consultants, and anyone else who needs to quickly extract important information out of Dentrix. Dentrix holds your data hostage. Dentrix Report Converter unlocks it. Currently only $49.99 for 7 days of access, and your results are guaranteed. Thanks!

Reduce Windows Defender’s scope

How many times has your computer come to a crawl, and when you open Task Manager it’s because the “Antimalware Service Executable” is running at 100%? This is Windows Defender, and it’s unstoppable. Literally, you cannot stop it. It gets to decide when to run, and often it feels like the best time is right in the middle of the day, when patients are in the chair.

To help reduce its impact, we’ve started to exclude a few applications from it, like so:

Control Panel > System and Security > Security and Maintenance > Virus Protection
Virus and Threat Protection Settings > Manage Settings
Add or remove exclusions
Add Process > MsMpEng.exe
Add Folder > C:\Program Files (x86)\Dentrix

Add any others that are getting caught in its net that run all the time (e.g. your imaging software).

Disable fetching of start menu web content

The Start Menu takes a long time to open because it fetches web content before popping up. To disable web search in the Start Menu, open PowerShell as an administrator and run this:

Set-ItemProperty -Path "HKCU:\SOFTWARE\Microsoft\Windows\CurrentVersion\Search" -Name "BingSearchEnabled" -Value 0 -Type DWord

Then reboot.

Source for this one.

Disable Fast Startup

This is a new one for us (at the time of this writing). A vendor was training the team on a new piece of software. The workstation they picked was running extremely slowly. She said, “have you restarted it and not just shut it down? I’ve been told that you have to restart it”. Then she proceeded to show that the machine claimed it had not been restarted in 2 weeks (when it had been shut down almost every night)!

After researching this, it turns out that, in an attempt to speed up Windows start, on some machines Windows will save the OS state on shut down, then reload that same state (memory leaks and all) on startup. This is called Fast Startup, and it’s been a part of Windows since Windows 8.

We’ve started to disable Fast Startup entirely. It’s a Control Panel > Power setting, and you can find screenshots about how to do so online. I’m not sure yet what kind of impact this will have, especially on those perniciously slow machines, but hopefully it will be a positive one!

I’ll continue to add and revise this as new tricks come up. If you have other ones, please feel free to comment.

How to install Dentrix 24.1 onto a remote workstation

For years now, whenever we swapped out a desktop and needed Dentrix reinstalled, our staff would call Dentrix Support to have them do the installation manually. This is because, sometime 5-10 years ago, when you double-clicked on “setup.exe” it just wouldn’t work. It often complained about missing components, .NET runtime versions, or similar. I think Crystal Reports was a frequent culprit. This was a far cry from when we bought the practice >15 years ago, when I could install Dentrix without much issue by putting in the CD(!) into the tray and running setup.

Programming note: If you need to convert Dentrix PDF reports to Excel, Google Sheets, or CSV, try Dentrix Report Convert. This is a simple service I’ve built to automate this. It is excellent for practice owners, office managers, accountants, attorneys, consultants, and anyone else who needs to quickly extract important information out of Dentrix. Dentrix holds your data hostage. Dentrix Report Converter unlocks it. Currently only $49.99 for 7 days of access, and your results are guaranteed. Thanks!

Today, I needed to get Dentrix set up on a remote computer to test some things. But I had no interest in calling Support to make this happen. I was determined, determined I say, to get this done myself.

Instructions

Note that these instructions were written for Dentrix 24 with Windows 10 Pro (10.0.19045 build 19405). I’m certain that they will break in the future. All of the steps below need to take place on the workstation.

  1. Navigate to \\[server]\DTXCommon\Installs\
  2. Copy the 24.1.1.17965 folder (or whatever is the latest one you see) to the workstation. I also copied the sibling files over to the Workstation as well, for good measure; not sure if that’s required or not. So, if this is what my server contained in \\[server]\DTXCommon\Installs\ :
    How Dentrix Server's Installs folder looks
    then this is what my workstation contained in my working folder:
    How the workstation's Installs folder looks, with only a subset of the Dentrix folder copied
    WARNING: even with the reduced number of files copied, this still required 3.2GB on disk.
  3. Try to run 24.1.1.17965\setup.exe . It will probably fail with an error about Crystal Reports.
  4. So, try to run Crystal Reports installation first. Find it in 24.1.1.17965\ISSetupPrerequisites\{CrystalReports-13-0-32bit-Prereq}\CRRuntime_32bit_13_0.msi . But, for me, this failed with Error 1904. Module C:\Program Files (x86)\SAP BusinessObjects\Crystal Reports for .NET Framework 4.0\Common\SAP BusinessObjects Enterprise XI 4.0\win32_x86\crtslv.dll failed to register. HRESULT -2147010895. Contact your support personnel. If it works for you and you don’t get this error, great; you can probably go back to step 3 and run the setup.exe for Dentrix again.
  5. After digging around the Internet, I found this answer on StackOverflow, which gave me the hint I needed. We have to reinstall the Microsoft Visual C++ 2015 Redistributable package. I found that on Microsoft’s website. Note: be sure to download the 32-bit version, because that’s what Crystal Reports relies on here apparently.
  6. When you try to install the Visual C++ 2015 Redistributable package, that may fail with an error if you already have one installed. I had to go to Settings > Control Panel > Add/Remove Programs, and uninstall Microsoft Visual C++ 2017 Redistributable (x64) 14.12.25810 first. After I did that, I rebooted, then worked backwards from here:
    • Installed Microsoft Visual C++ 2015 Redistributable Package (32-bit version), per line 5 above
    • Installed Crystal Reports via the CRRuntime_32bit_13_0.msi file, per line 4 above
    • Installed Dentrix via the setup.exe, per line 3 above

Other notes

Using Dentrix Workstation over a TailScale VPN is significantly slower than using Remote Desktop to the server or another workstation. There must be a ton of synchronous round trip calls made within Dentrix, with the (not invalid, at the time it was created!) assumption that your server and your workstations will all be connected to a high-speed intranet network.

Large content sites and SEO

I’ve always been fascinated with large content sites, domain authority, and long-tail SEO. The idea that one can create a lot of content targeting a specific niche, get that content indexed, and reap benefits of that investment over years through monetizing long-tail SEO, always seemed like magic.

The biggest barrier to entry was always that first part: creating high-quality content that targets a specific niche.

Today, after many years of percolating on this idea, I launched an experiment: a glossary of dental terms, hosted at dictionary.dental. The term list and definitions were built over time as a part owning the web presence for Forever Dental, with some assistance of large language models to flesh out some missing areas. The design and user interface are intentionally minimalistic, as I want this to be free and clear of ads and clutter. Hosting is intentionally incredibly simple. Not quite static-site, but heavily cached and running on a simple service. The glossary concept is incredibly repeatable too, so if this experiment works, it is scalable across niches.

And just like everything, it started very simply!

Relatedly, I found a fan page for Forever Dental. Turns out, people do love their dentist!

Why are release notes/change logs important?

I got involved in a Twitter/X conversation about whether release notes/change logs are important. It started with this post:

I can understand this sentiment from a consumer perspective. But after having spent 14 years delivering enterprise sales software, I believe release notes are undervalued at most software companies.

(Note: I’m using “release notes” and “change logs” interchangeably here.)

Release notes are important in enterprise software to the admin in charge of the software that you’ve sold to them. Because eventually, someone important above them (think: a VP or higher, often not directly in their reporting chain) will ask, “WTF! This software was working and now it isn’t. Admin, did the vendor change something?”

Admin then has to go figure it out. This is outside of their normal workday and set of responsibilities. Best case for the admin, they look at the change log, see that something did indeed change (and ideally for the better), and can copy/paste that back to the Important Person. Then they can go back to looking at Tiktok videos in between checking email.

The less good case for the admin is that they don’t find anything in the release notes. They then have to reach out to support/CSM and try to figure out what is going on. This takes time and adds to their work. They don’t like that.

If you can provide thorough, accurate release notes, this can increase the chance that the Admin hits the “best case” described above.

Some additional thoughts:

  • The more critical the software is in a revenue-generating workflow, the more watchful the admins will be to the vendor’s change logs/release notes
  • Many companies don’t bother with detailed App Store change logs, because they publish these directly in their product and away from competitors eyes. You wouldn’t believe how much critical information we gleaned from competitors about their product capabilities and core focus simply by watching what they published to their public change logs
  • Around 20-30% of our client admins overall watched our change logs at Mediafly. In some industries, that number was 100%

“The Bitter Lesson” and open source LLMs

Today I learned of a 2019 paper called “The Bitter Lesson

This paper postulates that, over the past 70 years, the biggest drivers of AI advancement were not due to special human-introduced nuances into what makes the model smart, but rather, dramatic advancement in computational resources driven by Moore’s Law (exponentially falling computational cost).

What does this mean?

Major AI leaps are driven not by companies rolling in a ton of “special sauce” into their AI models, but rather because it becomes dramatically cheaper to throw more hardware at the problem.

This gives me hope that the future of LLMs won’t be beholden to companies like OpenAI, Anthropic, and the like. But rather, we’ll see open source models catch up to and possibly surpass OpenAI’s GPT for raw text-to-text generation.

I am seeing that some open models claim to be on par with GPT 3.5, such as Mistral-7B and orca-2, but the empirical evidence is mixed. (p.s. hat tip to Anton Bacaj, who is a wealth of cutting-edge information around open models. https://twitter.com/abacaj )

Of course, there are other competitive areas where the private companies’ inherent advantages will allow them to dominate over open source models (marketing, enterprise features, APIs, wrapper support, stores, integrations, etc.). But at least the core offering won’t be wrapped up in a tight little, expensive, box.

Leaving Mediafly

Most of my writing these days is on LinkedIn and Twitter/X.

Cross-posted from LinkedIn:

Today marks the last day in my 14 year journey with Mediafly.

When I joined, I thought this would be a 2-4 year job. I joined as employee # 9, and have had the good fortune of participating in growing this company to hundreds of employees. I’ve experienced a whole career of experiences during that time, including helping sell and service to the world’s largest companies; building world-class Engineering and Product teams that rival some of the best I’ve ever worked with; completing and integrating 8 acquisitions; successfully guiding us through intense technology audits; and building scalable processes that will far outlast my tenure.

I met Carson, Mediafly’s founder and co-CEO, while at a previous startup where we did a joint project together. After a year of consulting for Mediafly, Carson and the team closed our first enterprise customer, RE/MAX, and I decided to join full-time. During these 14 years, Carson, your force of positive energy brought us through the good times and the bad.

I’m thrilled that Kelly Anderson is taking over as Chief Technology Officer. Kelly was the VP of Engineering for the largest of our engineering teams, and is fantastic with people and process. Kelly, Mediafly is so lucky to have you and for you to step into this role!

Carson, MaryJames, and the rest of the leadership team with whom I worked closely: you have a talented team at Mediafly. I’m so excited to watch where you all progress as I follow my path down this fork in the road. And to my friends I leave behind: I’m rooting for you all!

As for me, after taking a much needed break, I plan to dive back into entrepreneurship. I am exploring buying a small business, or starting one. Likely, but not necessarily, software. Stay tuned and stay in touch!

Dentrix services not starting on boot (2023 edition)

Last week, we experienced catastrophic failures with Dentrix. Every attempt to launch Office Manager, Appointment Book, Family File and Ledger, on the server and all workstations, crashed within seconds of launch.

Our system:

  • Dentrix 23.2.3.6311
  • Windows Server 2019 Essentials
  • 15+ workstations

We were without Dentrix for 6 hours while Dentrix Support uninstalled then reinstalled (!) Dentrix. That resolved it at the time, but it was not clear as to why it worked (and Support had no idea either).

Programming note: If you need to convert Dentrix PDF reports to Excel, Google Sheets, or CSV, try Dentrix Report Convert. This is a simple service I’ve built to automate this. It is excellent for practice owners, office managers, accountants, attorneys, consultants, and anyone else who needs to quickly extract important information out of Dentrix. Dentrix holds your data hostage. Dentrix Report Converter unlocks it. Currently only $49.99 for 7 days of access, and your results are guaranteed. Thanks!

A week later, Windows Update (KB5026362) downloaded on a Thursday, but did not yet install.

We restarted the server on the following Saturday and let the Windows Update complete. On Sunday, we attempted to launch Dentrix. Appointment Book and Office Manager would spin for a few seconds, then crash and shut down, on the server and on all workstations. Mild panic ensued. Interestingly, Document Center, Office Journal, and Timeclock still worked.

We attempted these things, none of which worked:

  • We re-enabled LLMNR
  • We uninstalled KB5026362

Lots of Google searching and deep thinking, we looked at Server Manager, and noticed that not all services were being launched at start (there was a red 5 next to it, indicating that 5 services did not launch). One of those was DtxUpdaterSrv. It listed status of “Stopped” but startup type of “Automatic”. Which means, it attempted to start but couldn’t. On a whim, I started the service manually, and immediately the broken applications started working

Digging deeper, I saw that these services, along with DentrixACEServer, all had attempted to start on system boot. None of them completed; every one of them had an Event log that showed “A timeout was reached (30000 milliseconds) while waiting for the … Service service to connect”.

With some searching, I came across this ServerFault post on how to change the default timeout from 30 seconds to 60 seconds. I applied the change and rebooted. The Dentrix services came up on the reboot, so things may be better! Only time will tell of course.

UPDATE: this seems to have worked, and we experienced no more issues.