(Un)Marked Mac OS
(Un)Marked Mac OS
The easiest way to perform an Undo on Mac is with a keyboard shortcut: Command + Z Simply press Command and Z together and the last action, typing, or activity will be “undone” on Mac OS. Note that some apps have multiple layers of Undo, meaning you can continuously Undo and revert back various steps and actions or typing. Connecting to cse.unl.edu and other unix servers with ssh from a Macintosh. Launch a terminal window. Type ⌘-K to launch spotlight. Type 'Terminal' and you should see.
The 'classic' Mac OS is the original Macintosh operating system that was introduced in 1984 alongside the first Macintosh and remained in primary use on Macs until the introduction of Mac OS X in 2001. Apple released the original Macintosh on January 24, 1984; its early system software was partially based on the Lisa OS and the Xerox PARC Alto computer, which former Apple CEO Steve Jobs.
I give up. After upgrading to OS X 10.9.3, I've experienced the least reliable computer of my life. I've had 16 kernel panics in 4 days, usually when plugging in remote displays, but sometimes just spontaneously.
In a meeting yesterday, a coworker exclaimed how reliable and awesome 10.9.3 was, and that he didn't have any kernel panics at all. I fantasized that his macbook spontaneously panic'd as he said that, proving him wrong in theatrical fashion, preferably with a loud bang and smoke. (It would also show that it wasn't just me.)
I settled for some debugging to see why his macbook was different, but he shooed me away from his keyboard as he was about to do an IMPORTANT DEMO, and didn't want me to jinx it. He plugged in a data projector.
Instant kernel panic.
He was now the third co-worker to experience this after upgrading to 10.9.3.
tl;dr: Jump to Update 8 for the workaround.
(unmarked Mac Os X
OS X saves diagnostic reports for each panic in /Library/Logs/DiagnosticReports. Note the dates:
These can be useful to browse for patterns. Maybe it's just one application which I can stop using?
No, it's completely random. This looks a lot like bad or misseated DRAM, so our helpdesk performed what they called a 'shell swap'. This means they replace everything except the SSD. Or put differently, they take out the SSD and put it in a 'known to be good' macbook pro, to see if the panics continue.
They continued.
Here's an example report:
So, CR2 (0x000000000000007a) looks bogus, leading to the panic. I can't infer much more than that, since I'm missing kernel symbols, and the stack trace is hex.
It would be nice if the kernel wasn't stripped, or, if it was easier to get the Kernel Debug Kit (please put them on opensource.apple.com). I'd help debug this further, but can't without symbols (at least, easily).
A few of us are now have the recurring 10.9.3 panics. After sending Apple many diagnostic reports with additional details, I'm switching back to 10.9.2. 10.9.3 is toxic with remote displays.
A coworker suggested the fix.
While I hope Apple fix this in 10.9.4, I'm now leery of OS X updates. I'd feel a lot better if I could debug this on my own further.
I should add that I've used Apple products and OS X for many years, and have been impressed by the reliability and quality of their work. 10.9.2 on the same laptop worked fine, and I'd have prior OS X releases with hundreds of days of uptime. I'd still recommend their products, as I hope this experience was an outlier.
Mac Os Download
Update 1
Symbols for this stack below (thanks Rasmus!)
Update 2
Based on the hackernews comments, some people have hit this and others haven't, and using external displays is a factor. Some have said that they have had issues with 10.9.2 as well, or all the 10.9 series. Perhaps it is a problem with a particular graphics card driver (myself and my coworkers have the retina Macbook Pros), but that's just a guess. The decoded stack above includes HFS calls, but that makes no sense, unless the graphics driver was stepping on random memory (which are among the worst panics to debug).
EDIT: After learning that this doesn't affect everyone, I changed the title of this post from 'Is Toxic' to 'Recurring Panics'. I also removed the words 'infected' and 'disease', which were off-putting.
Update 3
I got another new macbook pro, running 10.9.2 (although, a different kernel version), and switched using migration assistant. It worked great, initially. Then I had five panics in a row when connecting to different remote displays. In case migration assistant moved over some corrupted preferences, I got another new macbook pro, with 10.9.2, and just began using it fresh, and was still able to reproduce the panics (running only Firefox and Chrome, this time). I don't know if these panics are the same as what I had on 10.9.3, since I only have hex dumps to compare. 10.9.3 seemed to panic much more easily.
Mac Os Versions
I'm a little fed up of the typical macbook debugging technique: switch things until the problem goes away. I'm also fed up with seeing stack traces that are inscruitable hex. I want to read the stack traces, and understand what the kernel is doing that led to the panic. So I studied how Rasmus had translated my earlier stack, and I learned that the default kernel (mach_kernel) does have some symbols, which aren't used in the diagnostic reports. Excellent. I can write a quick helper tool for translating stacks.
Update 4
I wrote kernel_diagreport2text.ksh, a tool that translates symbols from OS X kernel diagnostic reports using two different techniques. It is here on github.
Here's my new 10.9.2 macbook panics, summarized by kernel_diagreport2text.ksh:
I wish I had this tool earlier! It uses atos(1) for symbol translation, and decorates remaining addresses with kernel extension names (eg, 'in com.apple.driver.AppleUSBHub') if available in the diag report. It does not need the Kernel Debug Kit installed, although if it is, you should get more symbols translated.
That output is for a default 10.9.2 system, and while many symbols are missing, we can still learn a lot. All of these panics are in VM, and don't look the same as the 10.9.3 panic that was translated earlier.
To run this yourself, download (or save) the raw script. Then open up Terminal (which is under Applications->Utilities) for a command line, and you can run it on your saved kernel diagnostic reports. The steps are likely something like this (depends where your browser has downloaded the file):
This script is (obviously) not an official Apple diagnostic tool, and is provided as-is with no warranties or guarantees. It does not need to be run as root.
Update 5
29-May-2014. I've partially translated my 10.9.3 panics, using my kernel_diagreport2text.ksh tool described earlier. Here are some key examples:
The others showed similar stacks. These are also all in VM, either page faults or an explicit panic() call in the VM_PAGE_QUEUES_REMOVE macro.
To translate my 10.9.3 diag reports with the kernel_diagreport2text.ksh script, I needed a copy of the 10.9.3 mach_kernel (I'm back on 10.9.2), and to edit the script to point to it (update: that's now the -f option). Apple's auto update had already downloaded the 10.9.3 update, putting it in /Library/Updates/031-02348/OSXUpd10.9.3.pkg. That pkg file turned out to be Russian dolls: a xar, containing a bzip2 file, containing a cpio archive, which contained the 10.9.3 mach_kernel.
If you ever want to do something similar yourself, you really want to make sure the mach_kernel matches what you have in the diag report, otherwise the translations will be incorrect. Eg:
Those match! The source for xnu-2422.100.13 isn't out yet, but when it is, it should be under opensource.apple.com/source/xnu. I've been browsing the earlier version to get a handle on the VM code.
Update 6
After a quick browse of the VM code, it looks like a double free, based on the VM_PAGE_QUEUES_REMOVE panic. It's possible the other VM panics are manifestations of the same bug. These are nasty bugs to debug, as the engineer must track down the earlier free. This is harder than it sounds, as free's occur so frequently. I could run out of memory trying to log them for later lookup.
But after using DTrace on the vm_page_free_prepare_queues path, I'm not sure it's a double free using the vm_* interface, as mem->free was not set. Which suggests something even nastier – someone else is stepping on memory, perhaps zero'ing it out. Now if two paths are fighting over the same memory, and if I'm lucky, they do so in either order with the 2nd hitting the panic. Which could mean I've already captured both paths in the earlier panics. The IOBufferMemoryDescriptor::free() could be the non-VM path that's freeing memory, coming from IOAcceleratorFamily2.
This is just speculation - I don't know what the real cause is yet. But given the nature of the panics (external displays), the partially-translated stacks, the open source xnu kernel, and DTrace to test theories, I have a lot of clues. The hardest part is finding time to put into this.
Update 7
Here's an older panic excerpt (OS X 10.5.8):
Notice something? This has arguments for the stack functions, which are incredibly useful when doing panic analysis, especially when all we have to go on is the panic report file. So where did these args go in 10.9?
From the xnu-2422.90.20 source, osfmk/i386/AT386/model_dep.c:
PRINT_ARGS_FROM_STACK_FRAME needs to be set. It's set in the same file:
(shakes fist.) No comment, no explanation, just zero that guy. Why?
I think the reason can be explained by an earlier kernel version, xnu-2050.9.2:
Ah. Stack frame arguments are architecture specific, and the code was written for i386, but not x86.
It should be a fairly easy enhancement for Apple to add the x86 code, and greately enhance all kernel panic reports.
Update 8
30-May-2014. I have a workaround in hand for Mavericks 10.9.2: turn off Firefox hardware acceleration. With this off, my panics have now stopped. Perhaps this works on 10.9.3 as well, if it is the same panic. The setting is under Preferences->Advanced->General.
Other applications use hardware acceleration as well, so you may need to disable it elsewhere, if you believe you have this bug. For me, the easiest way to duplicate the panic was to run a youtube video full screen in Firefox, then plug in a remote display, and close the laptop lid. A few cycles of that usually led to the same type of panic I was hitting earlier. And with hardware acceleration disabled, it worked fine.
I still feel this was much worse in 10.9.3 (if I had more time, I'd confirm). The 10.9.3 update did say that it 'Improves 4K display support', which may have modified the hardware acceleration routines.
I suspect hardware acceleration was freeing or stepping on memory it shouldn't, which led to the VM panics. I'd like to write more about this, including where I was in the analysis, and ideally identify the root cause, but now that I have a workaround it's no longer a priority to work on. I may do a follow up blog post later when I have the time. I wanted to share the workaround ASAP.
This information has been filed as Apple bug ID 17082120. (This is in addition to the 30 or so Kernel diag reports I've sent their way.)
(Un)Marked Mac OS