Category Archives: tech

Fixing Windows Server 2022 not launching services after reboot

Posted on by
Reply

We have been having an issue where, when a specific Windows 2022 server reboots (Windows update or some other reason), after reboot:

  • We cannot use Tailscale to get into the server
  • Dentrix is not running and therefore none of the client machines can access it
  • Hamachi (our legacy mesh network) weirdly still sometimes works

When we log in through Hamachi after this happens, sometimes as late as the next morning, the server looks like it is just finishing rebooting. Services start up that should have been running all night. Client machines are able to connect again.

This is obviously concerning, because if we weren’t there to log in, the network would be in a bad spot when the staff started their day.

After this happened several times over the past 6 months, I think I stumbled upon the specific issue and a possible solution.

Programming note: If you need to convert Dentrix PDF reports to Excel, Google Sheets, or CSV, try Dentrix Report Converter. This is a simple service I’ve built to automate this. It is excellent for practice owners, office managers, accountants, attorneys, consultants, and anyone else who needs to quickly extract important information out of Dentrix. Dentrix holds your data hostage. Dentrix Report Converter unlocks it. Currently only $49.99 for 30 days of access, and your results are guaranteed. Thanks!

Essentially, this is a common startup race condition with the Service Control Manager (SCM), especially in Windows Server 2022.
This usually happens when the service takes too long to initialize or depends on a system resource that isn’t ready during early boot. The presence of Event ID 7000 and 7009 confirm a startup timeout.
I verified a ton of Events with ID 7000 and 7009[1] right when the reboot took place, for critical services that should have restarted (notably, Tailscale and Dentrix).

To resolve, I

  • Open Registry Editor and go to: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control
  • Create or edit a DWORD value named: ServicesPipeTimeout
  • Set it to 180000 (decimal) for 3 minutes

Next I went to start > services.msc. For each service that had a 7000 or 7009 event, I switched its startup mode from Automatic to Automatic (delayed).

Then, when the staff wasn’t around, I reboot the server.

This appears to have addressed it! No more 7000 / 7009 errors, and I can successfully Tailscale into the server after a reboot.

[1]
To check for Event ID 7000 or 7009 (which indicate service startup timeouts) on Windows Server 2022, use the built-in filtering tools in Event Viewer:

  • Open Event Viewer: Click Start, type Event Viewer, and select Run as administrator
  • Navigate to System Logs: In the left-hand pane, expand Windows Logs and click on System
  • Filter the Log:
    • In the right-hand Actions pane, click Filter Current Log…
    • In the Event level section, check Critical, Warning, and Error
    • In the box labeled , type 7000, 7009
    • Click OK

“The Bitter Lesson” and open source LLMs

Posted on by
Reply

Today I learned of a 2019 paper called “The Bitter Lesson“

This paper postulates that, over the past 70 years, the biggest drivers of AI advancement were not due to special human-introduced nuances into what makes the model smart, but rather, dramatic advancement in computational resources driven by Moore’s Law (exponentially falling computational cost).

What does this mean?

Major AI leaps are driven not by companies rolling in a ton of “special sauce” into their AI models, but rather because it becomes dramatically cheaper to throw more hardware at the problem.

This gives me hope that the future of LLMs won’t be beholden to companies like OpenAI, Anthropic, and the like. But rather, we’ll see open source models catch up to and possibly surpass OpenAI’s GPT for raw text-to-text generation.

I am seeing that some open models claim to be on par with GPT 3.5, such as Mistral-7B and orca-2, but the empirical evidence is mixed. (p.s. hat tip to Anton Bacaj, who is a wealth of cutting-edge information around open models. https://twitter.com/abacaj )

Of course, there are other competitive areas where the private companies’ inherent advantages will allow them to dominate over open source models (marketing, enterprise features, APIs, wrapper support, stores, integrations, etc.). But at least the core offering won’t be wrapped up in a tight little, expensive, box.

Getting a Dell OptiPlex back up and running with Microsoft Windows 7

Posted on by
Reply

This post chronicles the month-long adventure in getting one of our Dell OptiPlex 3020M (you know, the tiny one with no CD/DVD drive) desktops back in good working order.

First, the problem:

  • Boot times start taking extremely long to get to the Windows loading screen, and then it would spin endlessly
  • Safe mode boot didn’t help this process along

As we were still under warranty with Dell, I reach out to them. Here is what happened:

  • I emailed Dell Support. They replied very quickly, and after a couple of days of back/forth with their tools, we isolated a bad hard drive.
  • I went through Dell Diagnostics with them (which is roughly baked into the BIOS). Turns out the hard drive was bad.
  • They sent me a replacement hard drive via FedEx Overnight.
  • I swapped it out and started the desktop. After booting to Windows and getting through a few prompts, I get stuck in a never-ending Dell configuration loop that is (what I call) the gray-on-black modal dialog of hell. Loop runs for 24 hours with no progress or hard drive activity. I can’t do anything (mouse and keyboard are disabled).

Dell OptiPlex 3020M stuck in gray on black loop of hell

Dell OptiPlex 3020M stuck in gray on black loop of hell

  • Dell Support directed me to download an .iso file that contains a preconfigured Windows 7 for my system. I didn’t know such a thing exists! Now I want to go get one for every one of the machines we maintain…
  • I download it, burn the .iso to USB with Rufus on the Surface Pro we have here (remember, no DVD ROM drive on the desktop I’m trying to recover), and try to get it to run.
  • I am able to boot to it and get through a few prompts. But very soon, I encounter a new error: “A required CD/DVD drive device driver is missing.”

"A required CD DVD driver is missing" dialog

“A required CD DVD driver is missing” dialog

  • I tried a number of things suggested by others on the Internet, including: switching the USB to another port midway through the installation, downloading Dell’s drivers and having Windows attempt to find the correct driver, buying an external CD/DVD writer, trying to burn the .iso to DVD (doesn’t work), and trying to just install Windows 10 from a later Dell machine onto this machine (also doesn’t work). Many hours were wasted.
  • I reach back out to Dell and complain about how much time I’ve wasted. After complaining, Dell Support sent someone onsite to replace the hard drive (again). After this was done, I was back at the neverending gray-on-black loop of hell.
  • I opened a new support ticket with Dell, this time around OS issues. I was sent new instructions for creating the Windows 7 Recovery USB key, which largely maps to this article. The article was very interesting, because:
    • It recommended using the diskpart Windows command line tool, vs. Rufus which I used previously, and
    • It didn’t quite work as is, as there was one significant discrepancy I discovered from those directions. I have a 64GB USB 3.0 key. I had to create a 16GB FAT32 partition (which is known to boot well; I can’t trust NTFS or ExFAT as boot partitions) vs. simply creating a full-disk 64GB partition. It took me a few tries to figure that out.
  • Success! This got me through the two key issues I had in the past (gray-on-black and missing device driver)
  • But, I encountered another problem: an error that said “Windows cannot be installed on this disk. The selected disk has an MBR partition table. On EFI systems, Windows can only installed on GPT disks.” Thankfully, this link indicated that I can safely delete all of the existing HDD partitions and create a new, big one. I did so, and the installation continues.

I wish the process wasn’t as difficult as it was. But at least I’m thankful that it works, and that my interactions with Dell Support helped me down the correct path, eventually. They were patient during the process, and had no problem sending their onsite tech to try to assist, which is the best I can hope for with business-level support.

Dear Enterprise Software Product Managers: Consider Scale

Posted on by
Reply

5425752305_7023df6048_o-1

Discussions about design and UX for software these days so often focus on onboarding. Scott Belsky, founder of Behance, even suggests A good discipline to help you stay simple is to focus at least 50% of your effort on onboarding and the first-time-user-experience. Providing a great onboarding flow is the quickest way for your users to find value in your new feature. After all, the sooner a new user is able to find value, the more sticky it’ll be for them, and the less churn you’ll experience, right?

Makes sense, and it gives a great starting point for how to think about a new feature. For example, from Mediafly’s point of view:

  • A newly signed-up user will start with 0 content, 0 salespeople, 0 users. Envisioning that scenario is very straightforward.
  • A small business might be using SalesKit by Mediafly to manage 50 pieces of sales collateral among 20 salespeople, distributed to 500 prospects. There is some complexity with this level of information, but for most features you might design, it’s probably pretty straightforward.

Often, however, this is where the design of a feature stops.

When you’re working with enterprise organizations with large user counts, diverse business processes, very large data sets, or whatever key metrics you track, however, you need to consider the user experience when there is high volume of use in these key metrics.

Example 1.

From the beginning, we designed Mediafly’s content management system (CMS), Airship, to start as simply as possible. From day 1, users could drag in content from their laptop, with reasonable defaults, and immediately get value. As our customers adopted our CMS and scaled out their use across diverse business processes and groups, we continued to discover issues that we could never have foreseen at launch.

Recently, a large customer (a major CPG enterprise)began uploading merchandising layout diagrams, hierarchically organized by region, for each of their tens of thousands of their customers’ stores to our system. This dramatically increased two key metrics: their volume of content (tens of thousands of new documents) and frequency of updates (thousands of changes every week).

Automating the upload and management process on their end is a no-go, as there is no common backend system where these documents reside. And, asking people to update these layout diagrams with our Airship CMS would require 20-40 hours a week of navigating, clicking, and dragging/dropping.

To address this, we conceived of a new upload model in which an adminstrator of a one of their region’s merchandising layout diagrams could organize the new content hierarchically on their laptop, zip up the file, and upload it into our system. We would then interpret the results and update the content automatically in the correct location. This solution at once solves both the problem of content volume and update frequency. And it can be reused for other customers who encounter similar challenges.

We spend as much time solving user experience challenges of scale as we do thinking about how to build compelling new features whose adoption will begin at very low volumes.

Example 2.

We recently released the ability for our content administrators to create special links to view content, which has been a hit with our Media/Entertainment customers. The link can have a password, be tied to a user account, or be public. Creating a link is straightforward, and initial reception and usage of this feature started off as very positive.

But, after a few months, we began to hear feedback from content admins about some challenges they were feeling as their use cases for links expanded. The volume increased dramatically in some of these use cases. We now see that some admins have to create as many as 200 individual links for individual users in a single day, usually around television pilots or key screening seasons.

After diving deeper into some of these workflows, we created a process diagram to show what the typical process is to create a link. The content admin:

  • Switches to their email client and composes a new email
  • Pastes in a template that they use for their emails
  • Switches to Airship
  • Finds content in the hierarchy
  • Navigates to the Links tab
  • Taps Create Link
  • Configures the link
  • Saves the link
  • Clicks the option to copy the link to the clipboard
  • Switches back to their email client
  • Pastes the link in
  • Sends the email

Whoa, that’s a lot of steps. Imagine having to go through this process 200 times in one day! For some admins, it requires the entire day.

We have since simplified the process to create a large number of links, and continue to improve upon the feature to solve the problem of high volume even further.

How these experiences have changed us

As we design new features, we now include an extra question to answer: What will this look like at high volume? At the design phase, we strive to have a hypothesis on how we would address problems of scale, and to see what we can do to simplify the initial UX even further should scale arrive faster than we can roll out a redesign.

However, just like most things we do from a product and engineering perspective, we operate iteratively. We certainly won’t prematurely optimize for scale. But by simply adding this question to our checklist of considerations, we’ve opened up the ability to solve the seemingly inevitable high volume issues that will arise.

(This post was cross-posted from the Mediafly Blog.)

Announcing SalesKit Meeting Tracker

Posted on by
Reply

(Cross-post from Mediafly and LinkedIn)

Here at Mediafly, we work hard to make the lives of salespeople better. We talk to a lot of salespeople, and what we hear from reps over and over again is “I hate my CRM.” This feeling comes from many reasons:

  • While CRM is very valuable to sales management, salespeople find these systems add little to no value to their primary job as salespeople
  • After a long day or week of meetings, salespeople have to drag themselves into their CRM to record notes of what happened in each sales meeting they had—instead, they want to be working towards their next deal or spending time with their families
  • CRM systems in general are clunky, complex, and require a lot of clicks/taps to accomplish the most basic of tasks

What does SalesKit Meeting Tracker do?
Once SalesKit Meeting Tracker has been turned on for your company’s environment, tap on the “Meetings” button and start tracking your meeting.

Start Meeting

When you’re done, stop tracking, record the meeting’s details, and send them to Salesforce.com.

 

Complete Meeting

Your meeting’s details will appear within the Activity History for the account, contact, and opportunity you’ve chosen.

What’s coming next?
We have a rich roadmap for SalesKit Meeting Tracker, including integration to our other app platforms (web, Windows, Mac, Android) and to other CRM platforms (Microsoft Dynamics, SAP Cloud for Customer).

Strategy is Culture

Posted on by
Reply

Mediafly in the Inc 5000

(Cross-posted from the Mediafly Blog)

When I used to envision how small companies create a great strategy, I would often imagine a single or multiple leaders sitting down and writing out something to the effect of “here are the three things that we’ll do to win.” The picture in my mind is complete with a boardroom or conference room, men and women wearing business suits, and hours of PowerPoint presentations at the ready. My experiences at a management consulting firm reinforced this: oftentimes, decisions started as hypotheses by the most senior leaders, and were then justified back to them by consultants who gathered and made sense of data.

After growing Mediafly by 657% over the past three years (and that’s not even including 2014 or 2015 numbers), I’ve come to shift my mental picture of how strategy is built at small companies. The big conference room is replaced by a small one, often with people on the other end of GoToMeeting; suits are replaced by jeans; PowerPoint is replaced by conversation with a few bits of data thrown in; and the confidence of “here are the three things that we’ll do to win” is replace by “here are what we think might be good to try, based on a little bit of data, a lot of intuition, and a willingness to adjust as we learn.”

But the biggest surprise in my mental picture comes from how we develop strategy. Our strategy is highly influenced and often derived by our incredibly empowering culture and system of ad-hoc conversations.

What makes our culture empowering?

At Mediafly, engineering, customer success, marketing, sales, and even interns have open, candid conversations with the executive team every day, passing along ideas and feedback in both directions. Customers can pick up the phone or write an email to anyone on the team, and expect to get help to solve issues. If team members find something they don’t like about any system, they are empowered to proactively fix it. Many team members work across divisional lines (engineering and marketing; sales and product; marketing and customer success), which empowers those team members to fix and grow even more.

What are ad-hoc conversations at Mediafly?

As a result of the culture, our team members find themselves in ad-hoc conversations about all parts of the business every day. Conversations will form at each others’ desks, over lunch or drinks, on our messaging systems (GChat, HipChat), and of course by email. These all serve to align how each of us thinks about our company’s place in the world. All of this coalesces and informs our company’s strategy.

A couple of great examples of what I mean:

Example 1:

Our strategy focuses on “content mobility”. Previously, this meant mobile (and primarily iOS) apps. However, several inputs came together to help us broaden and redefine this this past year:

  • Sales: We had several sales meetings in which it became clear that many companies simply won’t ever buy iPads for their sales staff
  • Product: After analyzing our usage data, we realized that the primary platform for using Mediafly is through our Web Viewer on a desktop browser (not a mobile device and not one of our apps)
  • Customer Success: A few conversations with existing customers crystalized that a desktop app would go a really long way to embracing Mediafly
  • Market: Windows Surface and touchscreen Windows 8 hybrid tablets are starting to gain traction

These inputs came in through our empowered team members thinking about their areas, via conversations at each others’ desks and at meals.

Assimilating these inputs was easy, when there were so many of them pointing in the same direction. As a result, we’ve begun to work on a desktop application and plan to release it within the next several months (stay tuned!).

Example 2:

We originally developed our Interactives Platform back in Q2 2012. At the time, we assumed Mediafly would build a consulting arm that caters to creating Interactives via consulting services to our enterprise customers, while our core team focuses on enterprise software licensing. We operated under this approach for almost a year, netting a reasonable amount of consulting work.

However, again, several inputs from our empowered team members, gathered via many conversations, helped reshape how we think about this:

  • Engineering and Product: Every time Engineering was pulled into an Interactive engagement during this time period, we had to delay work on the product side for a custom piece of work. However, the product side is where the bulk of our “enterprise value” would eventually come from.
  • Team: Getting sucked into consulting engagements of this form was extremely hard, and team members were working way longer than normal. This was largely because we weren’t used to the multiple back-and-forth design rounds required for consulting to overlap with product work, and time was tight.
  • Industry: we started to meet some great agencies and software development firms, and believed they could be incredible partners.

We made the decision in 2013 to terminate all new Interactives development for our customers (with a few exceptions). The result has been amazing: better products, better focus, happier customers, and partners who are doing amazing work.

As we continue to serve more customers with our phenomenal products and services, we will strive to:

  • Ensure team members continue to feel empowered to bring up ideas and issues,
  • Keep lowering friction for ad-hoc conversations to take place, and
  • Drive company strategy based on the output of our culture, as much as it is based on other external factors

A frustrating experience with returning to Google Drive from a Doc/Sheet/Slide

Posted on by
Reply

About a year ago, it became clear that Google Drive was attempting to focus on Docs, Sheets, and Slides as separate products. They launched separate apps for Docs, Sheets, and Slides on iOS and Android. It makes switching between a Doc and a Sheet more annoying. Maybe they are following the new trend in apps for a service to create a bunch of mini-apps (for example Foursquare’s recent split for the Swarm app).

But you know what’s really frustrating? Working with Google Docs/Sheets/Slides on the web. The service assumes that, because I am working on a spreadsheet now, the next document I work on MUST ALSO be a spreadsheet.

Google Drive Sheets upper left button

Here, I am working on a Sheet. But the next document I want to work on is a Doc. The only path out of this Sheet that I can see on the page is the giant green box next to the title of the Sheet. If I tap on that, I get sent to a list of other Sheets.

Sheets

 

Why, Google, why? Why don’t you send me to your beautiful, robust Google Drive home? There, I can get a view of everything I’m working on.

Instead, I find myself:

  • Opening a new tab
  • Typing drive.google.com
  • Then having to switch accounts (because, of course, Drive defaults me to my personal account and not my work account)

If someone over in the Googleplex can hear me, please consider changing this default behavior to point back to Drive.

IAM policy to allow full user access to an S3 bucket

Posted on by
Reply

Since I wrestle with this about once per month, here is the IAM policy to allow full user access to an S3 bucket.

{
   "Statement":[
      {
         "Effect":"Allow",
         "Action":[
            "s3:ListAllMyBuckets"
         ],
         "Resource":"arn:aws:s3:::*"
      },
      {
         "Effect":"Allow",
         "Action":[
            "s3:ListBucket",
            "s3:GetBucketLocation"
         ],
         "Resource":"arn:aws:s3:::bucket-name"
      },
      {
         "Effect":"Allow",
         "Action":[
            "s3:PutObject",
            "s3:GetObject",
            "s3:DeleteObject"
         ],
         "Resource":"arn:aws:s3:::bucket-name/*"
      }
   ]
}

Amazon Web Services and “cost stabilization”

Posted on by
Reply

When deleting a large chunk of data from Amazon S3, AWS struggles a bit with figuring out how much you actually owe.

We at Mediafly deleted ~30% of the tens of terabytes of data from one of our AWS accounts yesterday. One would expect the appropriate cost graph to drop ~30% as well. However, this (obfuscated) chart, output from our installation of Netflix Ice, the excellent open source tool contributed by Netflix, appears as if we outright deleted all content from our S3 bucket.

Amazon AWS S3 costs after deletion

 

Seeing this graph raised some alarms (“oh crap! Did we just delete everything??”), but thankfully the reality does not match what was depicted here. Time will tell when and how that information is updated, however.