“The Bitter Lesson” and open source LLMs

Today I learned of a 2019 paper called “The Bitter Lesson

This paper postulates that, over the past 70 years, the biggest drivers of AI advancement were not due to special human-introduced nuances into what makes the model smart, but rather, dramatic advancement in computational resources driven by Moore’s Law (exponentially falling computational cost).

What does this mean?

Major AI leaps are driven not by companies rolling in a ton of “special sauce” into their AI models, but rather because it becomes dramatically cheaper to throw more hardware at the problem.

This gives me hope that the future of LLMs won’t be beholden to companies like OpenAI, Anthropic, and the like. But rather, we’ll see open source models catch up to and possibly surpass OpenAI’s GPT for raw text-to-text generation.

I am seeing that some open models claim to be on par with GPT 3.5, such as Mistral-7B and orca-2, but the empirical evidence is mixed. (p.s. hat tip to Anton Bacaj, who is a wealth of cutting-edge information around open models. https://twitter.com/abacaj )

Of course, there are other competitive areas where the private companies’ inherent advantages will allow them to dominate over open source models (marketing, enterprise features, APIs, wrapper support, stores, integrations, etc.). But at least the core offering won’t be wrapped up in a tight little, expensive, box.

Leaving Mediafly

Most of my writing these days is on LinkedIn and Twitter/X.

Cross-posted from LinkedIn:

Today marks the last day in my 14 year journey with Mediafly.

When I joined, I thought this would be a 2-4 year job. I joined as employee # 9, and have had the good fortune of participating in growing this company to hundreds of employees. I’ve experienced a whole career of experiences during that time, including helping sell and service to the world’s largest companies; building world-class Engineering and Product teams that rival some of the best I’ve ever worked with; completing and integrating 8 acquisitions; successfully guiding us through intense technology audits; and building scalable processes that will far outlast my tenure.

I met Carson, Mediafly’s founder and co-CEO, while at a previous startup where we did a joint project together. After a year of consulting for Mediafly, Carson and the team closed our first enterprise customer, RE/MAX, and I decided to join full-time. During these 14 years, Carson, your force of positive energy brought us through the good times and the bad.

I’m thrilled that Kelly Anderson is taking over as Chief Technology Officer. Kelly was the VP of Engineering for the largest of our engineering teams, and is fantastic with people and process. Kelly, Mediafly is so lucky to have you and for you to step into this role!

Carson, MaryJames, and the rest of the leadership team with whom I worked closely: you have a talented team at Mediafly. I’m so excited to watch where you all progress as I follow my path down this fork in the road. And to my friends I leave behind: I’m rooting for you all!

As for me, after taking a much needed break, I plan to dive back into entrepreneurship. I am exploring buying a small business, or starting one. Likely, but not necessarily, software. Stay tuned and stay in touch!

Dentrix services not starting on boot (2023 edition)

Last week, we experienced catastrophic failures with Dentrix. Every attempt to launch Office Manager, Appointment Book, Family File and Ledger, on the server and all workstations, crashed within seconds of launch.

Our system:

  • Dentrix 23.2.3.6311
  • Windows Server 2019 Essentials
  • 15+ workstations

We were without Dentrix for 6 hours while Dentrix Support uninstalled then reinstalled (!) Dentrix. That resolved it at the time, but it was not clear as to why it worked (and Support had no idea either).

A week later, Windows Update (KB5026362) downloaded on a Thursday, but did not yet install.

We restarted the server on the following Saturday and let the Windows Update complete. On Sunday, we attempted to launch Dentrix. Appointment Book and Office Manager would spin for a few seconds, then crash and shut down, on the server and on all workstations. Mild panic ensued. Interestingly, Document Center, Office Journal, and Timeclock still worked.

We attempted these things, none of which worked:

  • We re-enabled LLMNR
  • We uninstalled KB5026362

Lots of Google searching and deep thinking, we looked at Server Manager, and noticed that not all services were being launched at start (there was a red 5 next to it, indicating that 5 services did not launch). One of those was DtxUpdaterSrv. It listed status of “Stopped” but startup type of “Automatic”. Which means, it attempted to start but couldn’t. On a whim, I started the service manually, and immediately the broken applications started working

Digging deeper, I saw that these services, along with DentrixACEServer, all had attempted to start on system boot. None of them completed; every one of them had an Event log that showed “A timeout was reached (30000 milliseconds) while waiting for the … Service service to connect”.

With some searching, I came across this ServerFault post on how to change the default timeout from 30 seconds to 60 seconds. I applied the change and rebooted. The Dentrix services came up on the reboot, so things may be better! Only time will tell of course.

Escaping the present

From an external news perspective, 2017 has been a difficult year. I find the current administration to be full of dangerous nitwits who will either get us all killed or leave our way of life severely weakened.*
I found myself continually sad at the end of each night during the first quarter. I would read the news or browse Facebook (which consisted mostly of news for me) right before going to bed, then be depressed at what the next four years would bring.
Come Q2, and specifically a long weekend trip we planned to take in early June, I felt an urge to read fantasy/sci-fi. I recalled some elements of Brandon Sanderson’s takeover of The Wheel Of Time series, but felt like the last three books were worth another read. And so, I picked up each of books 13, 14, and 15, and read them at nights, during free time, and more.
I didn’t stop there. After plowing through them in 2 months, I then decided to re-read Sanderson’s The Stormlight Archive series. I finished those two books in another month.
And looking back, even though there were moments where I was definitely overdoing it (somewhat ignoring my family, sneaking pages in here and there), I was way, way happier falling asleep than when I read the news or Facebook.
So, here’s to hoping I can find a happy medium!
*Every other part of my life has been wonderful. I guess I can’t complain too much.

Getting a Dell OptiPlex back up and running with Microsoft Windows 7

This post chronicles the month-long adventure in getting one of our Dell OptiPlex 3020M (you know, the tiny one with no CD/DVD drive) desktops back in good working order.

First, the problem:

  • Boot times start taking extremely long to get to the Windows loading screen, and then it would spin endlessly
  • Safe mode boot didn’t help this process along

As we were still under warranty with Dell, I reach out to them. Here is what happened:

  • I emailed Dell Support. They replied very quickly, and after a couple of days of back/forth with their tools, we isolated a bad hard drive.
  • I went through Dell Diagnostics with them (which is roughly baked into the BIOS). Turns out the hard drive was bad.
  • They sent me a replacement hard drive via FedEx Overnight.
  • I swapped it out and started the desktop. After booting to Windows and getting through a few prompts, I get stuck in a never-ending Dell configuration loop that is (what I call) the gray-on-black modal dialog of hell. Loop runs for 24 hours with no progress or hard drive activity. I can’t do anything (mouse and keyboard are disabled).

Dell OptiPlex 3020M stuck in gray on black loop of hell

Dell OptiPlex 3020M stuck in gray on black loop of hell

  • Dell Support directed me to download an .iso file that contains a preconfigured Windows 7 for my system. I didn’t know such a thing exists! Now I want to go get one for every one of the machines we maintain…
  • I download it, burn the .iso to USB with Rufus on the Surface Pro we have here (remember, no DVD ROM drive on the desktop I’m trying to recover), and try to get it to run.
  • I am able to boot to it and get through a few prompts. But very soon, I encounter a new error: “A required CD/DVD drive device driver is missing.”

"A required CD DVD driver is missing" dialog

“A required CD DVD driver is missing” dialog

  • I tried a number of things suggested by others on the Internet, including: switching the USB to another port midway through the installation, downloading Dell’s drivers and having Windows attempt to find the correct driver, buying an external CD/DVD writer, trying to burn the .iso to DVD (doesn’t work), and trying to just install Windows 10 from a later Dell machine onto this machine (also doesn’t work). Many hours were wasted.
  • I reach back out to Dell and complain about how much time I’ve wasted. After complaining, Dell Support sent someone onsite to replace the hard drive (again). After this was done, I was back at the neverending gray-on-black loop of hell.
  • I opened a new support ticket with Dell, this time around OS issues. I was sent new instructions for creating the Windows 7 Recovery USB key, which largely maps to this article. The article was very interesting, because:
    • It recommended using the diskpart Windows command line tool, vs. Rufus which I used previously, and
    • It didn’t quite work as is, as there was one significant discrepancy I discovered from those directions. I have a 64GB USB 3.0 key. I had to create a 16GB FAT32 partition (which is known to boot well; I can’t trust NTFS or ExFAT as boot partitions) vs. simply creating a full-disk 64GB partition. It took me a few tries to figure that out.
  • Success! This got me through the two key issues I had in the past (gray-on-black and missing device driver)
  • But, I encountered another problem: an error that said “Windows cannot be installed on this disk. The selected disk has an MBR partition table. On EFI systems, Windows can only installed on GPT disks.” Thankfully, this link indicated that I can safely delete all of the existing HDD partitions and create a new, big one. I did so, and the installation continues.

I wish the process wasn’t as difficult as it was. But at least I’m thankful that it works, and that my interactions with Dell Support helped me down the correct path, eventually. They were patient during the process, and had no problem sending their onsite tech to try to assist, which is the best I can hope for with business-level support.

A dead simple way to improve your writing

Why does anyone think using the phrase “dead simple” is remotely acceptable?

Example:

DishwasherNow is the dead simple way to have a brand new dishwasher delivered to your door from iPhone.

Is “dead” an adjective for “simple”, which has morphed into a noun but has also ceased functioning in this world?

Or is this statement implying that the service is so easy to use that the walking dead can even order their dishwasher from the iPhone they are carrying in their lifeless hands?

Why can’t we just use “easy” or “simple”?

DishwasherNow is the easy way to have a brand new dishwasher delivered to your door.

See how much better that reads? You don’t have to be a rockstar engineer or ninja growth hacker to understand what this phrase means.

EDIT: I’m on the losing end of this battle (from Google Books Ngram Viewer):

Dead Simple usage in Google Ngram Viewer

A practiced method to solve hard problems

Here at Mediafly, we are faced with hard engineering, product, sales and marketing problems every day. Each of us takes a different approach to solving these problems. Some of us like to create pros/cons lists. Others dig deep into data and use that help answer every question. No one approach is the “right” approach for everyone.

I recently had a conversation with our Engineering Manager[1], and he described his approach to solving hard problems.

  • Take an attempt to solve the problem, but don’t stress about it if you can’t figure out the solution yet.
  • Review the key aspects of the problem right before you go to sleep. This involves working on the problem from multiple angles. Meaning, if you tried one solution and it doesn’t work, try another. If it did work but it’s ugly, just note the key parts of why it works and why it’s ugly. You need to get intimate with the problem and be really familiar with it from all angles.
  • Now, let the problem go. Sleep on it, take a shower, go for a run. Do something to take your mind off of it entirely.
  • When you least expect it, an insight will find you. When the solution does find you, immediately explain it to as many people as you can. Don’t worry about whether they are an expert in the subject domain. Just start explaining. The mere process of explaining acts as a forcing mechanism to refine the solution further. It also serves as a filter; if what you thought was initially a great idea turns out to not be, attempting to explain may allow you to filter out the seemingly-good idea much more quickly, and get back to solving the problem another way.

I’ve watched him apply this method of problem solving over the years, and it truly is a thing of beauty. He will often take 2, 3, 4 attempts at particularly thorny engineering problems. He will sometimes throw away the code he wrote for an attempt and go back to the drawing board. He will restart this process from scratch as necessary. But, regardless, he almost always comes up with a solution that solves the problem elegantly. And watching his success has led me to begin adopting this approach for problems of all sort that I face as well.

[1] Special thanks to @laimis for being the inspiration for this method and this post! And he credits A Technique for Producing Ideas as inspiration for this process.


Dear Enterprise Software Product Managers: Consider Scale

5425752305_7023df6048_o-1

Discussions about design and UX for software these days so often focus on onboarding. Scott Belsky, founder of Behance, even suggests A good discipline to help you stay simple is to focus at least 50% of your effort on onboarding and the first-time-user-experience. Providing a great onboarding flow is the quickest way for your users to find value in your new feature. After all, the sooner a new user is able to find value, the more sticky it’ll be for them, and the less churn you’ll experience, right?

Makes sense, and it gives a great starting point for how to think about a new feature. For example, from Mediafly’s point of view:

  • A newly signed-up user will start with 0 content, 0 salespeople, 0 users. Envisioning that scenario is very straightforward.
  • A small business might be using SalesKit by Mediafly to manage 50 pieces of sales collateral among 20 salespeople, distributed to 500 prospects. There is some complexity with this level of information, but for most features you might design, it’s probably pretty straightforward.

Often, however, this is where the design of a feature stops.

When you’re working with enterprise organizations with large user counts, diverse business processes, very large data sets, or whatever key metrics you track, however, you need to consider the user experience when there is high volume of use in these key metrics.

Example 1.

From the beginning, we designed Mediafly’s content management system (CMS), Airship, to start as simply as possible. From day 1, users could drag in content from their laptop, with reasonable defaults, and immediately get value. As our customers adopted our CMS and scaled out their use across diverse business processes and groups, we continued to discover issues that we could never have foreseen at launch.

Recently, a large customer (a major CPG enterprise)began uploading merchandising layout diagrams, hierarchically organized by region, for each of their tens of thousands of their customers’ stores to our system. This dramatically increased two key metrics: their volume of content (tens of thousands of new documents) and frequency of updates (thousands of changes every week).

Automating the upload and management process on their end is a no-go, as there is no common backend system where these documents reside. And, asking people to update these layout diagrams with our Airship CMS would require 20-40 hours a week of navigating, clicking, and dragging/dropping.

To address this, we conceived of a new upload model in which an adminstrator of a one of their region’s merchandising layout diagrams could organize the new content hierarchically on their laptop, zip up the file, and upload it into our system. We would then interpret the results and update the content automatically in the correct location. This solution at once solves both the problem of content volume and update frequency. And it can be reused for other customers who encounter similar challenges.

We spend as much time solving user experience challenges of scale as we do thinking about how to build compelling new features whose adoption will begin at very low volumes.

Example 2.

We recently released the ability for our content administrators to create special links to view content, which has been a hit with our Media/Entertainment customers. The link can have a password, be tied to a user account, or be public. Creating a link is straightforward, and initial reception and usage of this feature started off as very positive.

But, after a few months, we began to hear feedback from content admins about some challenges they were feeling as their use cases for links expanded. The volume increased dramatically in some of these use cases. We now see that some admins have to create as many as 200 individual links for individual users in a single day, usually around television pilots or key screening seasons.

After diving deeper into some of these workflows, we created a process diagram to show what the typical process is to create a link. The content admin:

  • Switches to their email client and composes a new email
  • Pastes in a template that they use for their emails
  • Switches to Airship
  • Finds content in the hierarchy
  • Navigates to the Links tab
  • Taps Create Link
  • Configures the link
  • Saves the link
  • Clicks the option to copy the link to the clipboard
  • Switches back to their email client
  • Pastes the link in
  • Sends the email

Whoa, that’s a lot of steps. Imagine having to go through this process 200 times in one day! For some admins, it requires the entire day.

We have since simplified the process to create a large number of links, and continue to improve upon the feature to solve the problem of high volume even further.

How these experiences have changed us

As we design new features, we now include an extra question to answer: What will this look like at high volume? At the design phase, we strive to have a hypothesis on how we would address problems of scale, and to see what we can do to simplify the initial UX even further should scale arrive faster than we can roll out a redesign.

However, just like most things we do from a product and engineering perspective, we operate iteratively. We certainly won’t prematurely optimize for scale. But by simply adding this question to our checklist of considerations, we’ve opened up the ability to solve the seemingly inevitable high volume issues that will arise.

(This post was cross-posted from the Mediafly Blog.)

Announcing SalesKit Meeting Tracker

(Cross-post from Mediafly and LinkedIn)

Here at Mediafly, we work hard to make the lives of salespeople better. We talk to a lot of salespeople, and what we hear from reps over and over again is “I hate my CRM.” This feeling comes from many reasons:

  • While CRM is very valuable to sales management, salespeople find these systems add little to no value to their primary job as salespeople
  • After a long day or week of meetings, salespeople have to drag themselves into their CRM to record notes of what happened in each sales meeting they had—instead, they want to be working towards their next deal or spending time with their families
  • CRM systems in general are clunky, complex, and require a lot of clicks/taps to accomplish the most basic of tasks

What does SalesKit Meeting Tracker do?
Once SalesKit Meeting Tracker has been turned on for your company’s environment, tap on the “Meetings” button and start tracking your meeting.

Start Meeting

When you’re done, stop tracking, record the meeting’s details, and send them to Salesforce.com.

 

Complete Meeting

Your meeting’s details will appear within the Activity History for the account, contact, and opportunity you’ve chosen.

What’s coming next?
We have a rich roadmap for SalesKit Meeting Tracker, including integration to our other app platforms (web, Windows, Mac, Android) and to other CRM platforms (Microsoft Dynamics, SAP Cloud for Customer).