Hey everyone,

So, as it turns out, I haven’t been using this blog very much. Sorry 🙁 Also, this is a relatively personal post and may contain grammatical errors (I’m tired).

Oh man, 2013. Probably the strangest year for me so far. I spun my wheels for a long time trying to make money doing things I’m frankly not that good at. Better than average maybe, but not exceptional. Then for the last quarter I started contracting within my actual trained skillset.

Honestly, it felt good to do what I’m good at again. It was a big step for me as well, I had gotten into the mindset that I may have just burned out of the industry. I’m glad I didn’t, I just needed a break. The contracting ended up taking almost all of my time those months. I’m still contracting, but managing my time a little better so far this month. I’m actually able to do something other than work and sleep :/

Normally, I don’t bother with the whole “New Year’s Resolutions” stuff. And honestly, I didn’t set any resolutions this year either. Resolutions are dumb (post about that incoming later). Goals are actually useful. I took enough time off around the holidays to actually get a little creative and think laterally again. So, let’s get into the meat of it, shall we?

2014 Goals

I made a list of 22 goals for 2014, broken into two lists of 11. The first 11 are my goals, the second 11 are stretch goals or minor goals. I had decided that I should make sure my goals fit into the following categories:

  • Health
  • Lifestyle
  • Career/Professional Goals

Yes, health and lifestyle overlap quite a bit, and honestly I’m not sure I did a good job filling out the “lifestyle” part, unless you consider things that may not contribute to my career as “lifestyle.” Maybe? I don’t know, but it was just a guideline.

I normally don’t post this sort of stuff publicly, but hey, maybe I’ll get farther along if I post it. The deadline for all of these is the end of the year, though I kind of hope I hit some of them sooner.

1. Lose 60 pounds

I need to lose weight. More than 60 lbs, that’s for sure, but this seems like an attainable goal. That averages to only a little over 1 pound / week. I have this written down as weighing under a specific weight on my own goal list here, so I can’t get away with that whole “I lost 10 pounds” then gain it back and more, then start over and “lose 10 pounds” and so on. I swear, some people lose 60 pounds every year, but somehow are heavier than last year at they physical.

Anyways, this is simple enough in concept, and I can’t cheat like that. I have ideas on how to accomplish this, but that’s for another post. Moving along.

2. 2 full novels published (self-published is fine)

No more excuses on the writing, guys. I want to make sure I have 2 novels written and available for people to read, preferably by purchasing it from Amazon or something. I might think about posting on something like fictionpress along the way. It’s scary, but I think that’s the first step towards putting something out on a paid service and dealing with negative reviews and old fashioned goodreads vitriol.

What’s a “full” novel? I’m aiming for 250 pages or more per book. No less than 200 pages.

What is “published”? No, fictionpress or this blog doesn’t count. It has to be online as an ebook that someone needs to pay for to read. Amazon, Barnes & Noble, or something similar.

3. 3 volumes of the light novel published (illustrated, hard copies)

I’m working on a light novel series right now. It’s not very far along, I just kind of know the main characters and the basics of the world at the moment. I know it’s a light novel series though, and I have some weird ideas about publishing. Anyways, when I have enough for a volume (200 pages or s0), I want to get an illustrator lined up, and have hard copies available. Even if I only make, like, $1 per copy, I want to do this.

This is quite a bit harder than the novels. More writing, more research, and more production.

If you’re keeping track, we’re at a minimum of 1000 pages written.

4. 1 manuscript submitted to an agent or publisher

Another novel, probably. I want to get in a professional relationship with a literary agent, if I can. I know that acceptance is near impossible at first, and that’s OK. There’s one piece of phrasing here that makes harder than it seems. At first glance you’d think “oh, you just need to send it in, that’s no problem.” Not quite. First, you have to send a synopsis and query letter. After you get some interest, they might ask for the manuscript.

I want to give this the old college try, though with how long the backlogs for these people can be, it’s entirely possible I send out dozens of query letters and get no responses before the year’s out. I don’t already have the connections in the industry, so I’d be in the queue if I didn’t end up in the circular file right away.

(1200 pages now)

5. 50 new posts published here

Here, meaning this blog. This goals is what got me back writing on here. I know this post sucks, but I have to get back into the groove.

I hope to write a few technical posts (the one technical post I wrote is easily the most popular), some more of these, and fill out some other types of content. I’m too obsessive about quality to churn out 50 short or garbage posts, so this could end up taking a while.

6. 1000 followers on Twitter (@KadonoGaming)

At least, probably @KadonoGaming. I also want to get @MattEkenstedt patched up and start posting there again. I know dividing my attention and the people who follow me like that is a bad idea, but I can’t really see @KadonoGaming being a useful Author/Productivity profile. Anyways, the goal isn’t for the combined total to be 1000. No, one of the individual accounts needs to have over 1000 followers. That’s not really that hard of a number to get. I don’t think Twitter is really the marketing platform that it keeps being heralded as. I doubt many people find you on Twitter, then follow you to your other platforms. But hey, who knows, right? And having at least 1000 followers is necessary to have a respectable Twitter account.

7. Taken the JLPT N5

I actually have this listed as “Passed the JLPT N5” but since I’m going to take it in December, and you don’t find out your pass/fail status until the next year, I can’t really know if I passed or not. Passing is the goal, obviously, but I won’t know in 2014. And before anyone jumps on me for taking the N5, I know it’s basically useless for your career and so on. I could probably take the N4 this year and have a chance at passing (Kanji being my greatest weakness there). My reasoning for starting at the bottom is that it gives me an opportunity to feel out how the test environment/process is handled and it’s a first step. I don’t have any formal education in Japanese either, so I think it’s best for me to start at the bottom.

8. 300 Twitch followers

Again, I’m trying to increase my reach here. Right now I have 88, which isn’t bad, but it isn’t good either. I got 87 of those last year, so I figure with the snowball effect, 300 should be easy. The main limiting factor is that I’m not a part of a network.

Note: Things are getting weird around game streaming and game videos. Hopefully this doesn’t mess me up too much.

9. $10k income from non-salary/contract/hourly source

Since I’m trying to diversify my income sources (and reduce risk), I wanted to put a number on it. $10k isn’t all that much over the course of a year. Especially since this is income and not revenue. $10k revenue would be a much harder goal. At any rate, if I can drum up at least this much it will be a sign that I’m successfully diversifying my income.

10. 100 RSS subscribers on a new podcast (or on YouTube)

New Podcast? Yeah, new podcast. Specifically, I need to produce non-live content for people to consume. 100 is incredibly low if we’re talking about YouTube, but with all the ContentID stuff on YouTube I might keep away from there. It seems like if you’re not already big, stuff like gaming content will get your channel shut down once you gain some traction.

100 RSS subs without already having an audience is actually kind of a challenge. Especially since people tend to go through YouTube for this stuff. The final reason I don’t have too ambitious of a number here is that I just don’t know exactly what I’m going to do yet, so I want to keep my goal attainable.

11. 10 new plushies designed and created

No chance I can make a living out of it, but I want to keep doing this because I think it takes me out of my normal few circles enough that it keeps me more creative. I want to do a few pokemon, maybe some food or something, we’ll see.

Bonus Goals

No talking about these individually, just a list:

12. Completed the Genki Textbook
13. Made over $7000 gross income one month
14. Founded a publishing company LLC
15. Lose 100 lbs
16. 5 manuscripts completed
17. YouTube channel (or RSS feed) with 500 subs
18. Podcasts or regular shows (each running for 3 months or more)
19. $3000 through patreon (or similar)
20. 5 guest spots on blogs (or podcasts or videos)
21. 100 people on a mailing list of some kind
22. 100 new posts on this blog

Most of those are related to the first 11. Either related, ambitious goals, or just a straight up stretch goal for an earlier goal.

Summary

If you’ve gotten this far, thanks. If you have, you probably noticed that I put a lot of focus on writing, mostly in novel form, but also this blog. In fact, almost everything is about content creation.

I basically stopped making content for about 3 months there. Honestly, I hated it. After doing content creation of some kind for a couple years, it was horrible to feel so creatively drained. The lack of content is entirely my fault. Poor time management, and even poorer energy management.

Some of the goals are money oriented, but none of the numbers are all that big if you think about the time frame. They’re just there to provide a progress marker. If I can make $10k over the year on things aside from my primary income, I’m doing the right things. People are engaged and think my content is worth a few bucks.

Let me be clear with you guys. My long term goal here is to get a content publishing company going that involves more than just me (hopefully people much more talented than me). The only way I can think of to bootstrap this without industry connections or lots of money is by starting with my own stuff. Once I’ve gotten familiar with what’s out there and have some traction, I can bring in other people. To be able to do that, though, I’ll have to be able to pay those people.

So I think about it like this: These projects I’m working on are like a car starter. I spin as fast as I can, and some of the rotations might seemingly do nothing, but eventually the engine will start. That’s the theory anyways.

This post was little more rambling than I was hoping, but it’s a personal post, not an instructional one. I wanted to put my goals out there. Maybe I’ll accomplish all of them, but probably not. Still, I expect to accomplish more than half of them and be much farther along by the time the year ends.

It’s going to be a productive year, I can feel it.

I am not a minimalist. I’m no hoarder, but I’m certainly not a minimalist.

The minimalist’s lifestyle is something I’ve always been tangentially interested in. I look at people who have all of their desktops and counters perfectly clean and beautiful, who spend their days in thought and consideration and I’m a little jealous. I’m hoping to outline some ideas I have regarding my own first steps into something resembling minimalism.

Motivation

There have been a number of motivations over the years, but here are a few that have been weighing on me in recent months:

  • All of the unpacked boxes – Half my garage and an entire room in my basement are filled with things that haven’t been unpacked since I moved here. Actually, the stuff in the room has been unpacked and I had one real attempt at organizing it, but that’s where it ended.
  • A stack of empty boxes – I have a stack of empty product boxes taller than I am. Some of them were kept for ease of moving, some were just kept ‘for the warranty period’ which has long since lapsed, and I have no idea why I kept the others.
  • I could use some money – It’d be nice to turn some percentage of the stuff I have sitting around into cash. It’s not like any of it will get more valuable.
  • Clutter – I just have too much clutter. Everywhere. IT’s one thing to have little knick-knacks around, but when all of the flat surfaces in every room have things on them it’s time to do something.
  • A hundred little things – Honestly, there are so many little reasons I’m getting frustrated by the over-consumption attitude of society right now.

Philosophical Changes

Another thing that’s happened over the last several years is a large philosophical change in myself. For a handful of years (right after college) I decided that the best thing was to have everything. I had these ideas about having an expensive car and I started to buy. I upgraded my computer, I bought all sorts of video games (and played very few of them), and so on. I had stuff and I wanted people to see the stuff that I had, so I’d invite friends and coworkers over to use my stuff.

Looking back, I find that I’m not exactly disgusted like some people might be. I was naive and wasteful. The joy I got from things was misplaced. Here’s an example: I used to shop on woot.com all the time. I don’t have a particular problem with the service they offer and some of the things they sell are actually pretty good deals. The problem was that I would buy these things on woot, not because I wanted the item, but because I wanted to buy the item. The best example of this that I can think of was when woot was selling these 20 packs of WD-40 grease pens. One of those pens could be useful if it was cheap, and maybe even 20 if I wanted to give them to family members. I bought 60 (Buy Three!). I got a kick out of having bought a ridiculous number of these grease pens and the chuckle it got out of coworkers. Then it got to my apartment, I gave out maybe a half dozen of them, used one once, and had two unopened 20 packs sitting in my closet. I still haven’t opened them. I wonder if they go bad.

Clearly I didn’t need or even really want the items, but back then I didn’t see any issue with that.

Today I don’t go to woot.com. I know that it’s bad for me and causes me to buy for the joy of buying, which isn’t a very fulfilling reason. A good chunk of the reason I want to move towards minimalism is to give me pause before I buy something. That way, hopefully, I’ll  only buy the things I actually need or that would be genuinely entertaining.

Where to Begin

As I implied before, I’ve read a lot about minimalism. To summarize what I remember about what I’ve read, people have done some of the following:

  1. Give away everything – An extreme route to be sure. Usually there’s something deeper here, like a religious or philosophical conviction.
  2. Sell (nearly) everything – The slightly more practical version of the above. The idea here is to just get rid of everything you don’t actually need, and use the money to pay off debt or fund a lifestyle.
  3. Surface purge – This is where you start by making it so that all of your flat surfaces have nothing on them that doesn’t have to be there. It’s obviously only a first step, but it makes sense to me.
  4. Empty the closet – A logical starting point, since we all have clothes we’re not actually going to wear ever again.
  5. Declutter a few things a day – The slow and steady route. Good as long as you can develop the habit.

Essentially it boils down to two or three methodologies. Go bold, be visible, and do what you can. They aren’t all mutually exclusive, though generally the people who do the top two aren’t going to do it one at a time. The third and fourth are about quick, visible change to drive motivation.

Where I’ll Begin

6972571733_7ed9b204dd_z

As we’re all kind of different from each other, but all kind of the same; I’ll be doing something that’s both different and the same. I’m not nearly bold enough to just get rid of everything. I should really outline what I want to get out of this experiment first:

  1. Spend less
  2. Spend intelligently
  3. Have less clutter
  4. Get some money back out of my current clutter

Emptying out my closet won’t accomplish much of that. Less clutter, but that’s about it. It needs to be done, but I think it can wait. Similarly basic decluttering or a surface purge wouldn’t accomplish many of these goals.

That’s why I’ve decided to start with food. I have probably two months or more worth of non-perishable and relatively non-perishable food around the house. More if I get into basic staples. For my own resource I’m going to outline what I know I have sitting around:

  • At least 10 lbs of dry rice
  • An entire freezer of meat (mostly raw)
  • Another freezer of meat (I think)
  • A freezer filled with boiled and bagged beans (in 1/2 and 1 lb bags)
  • Some canned vegetables
  • Other random pantry foods

Most of what I have by volume is rice, meat, and beans. I’m not sure this is the healthiest way to eat, since I’d be eating entirely grains and proteins if I bought nothing. So here are the rules:

  1. For the first two weeks, I don’t get to buy any food.
  2. For weeks 3 and 4 I get $20/week to spend on groceries.
    1. This food has to be either fresh vegetables or fruit (fresh or canned)
    2. The budget is the upper limit and it expires at the end of the week.
  3. Eating out is allowed in the following cases:
    1. Planned outings with friends – budget is $20/week
    2. When I can’t avoid it
  4. If at any time I can tell that I’m getting seriously malnourished, any rule can change.

These rules only go out one month and I plan to write up another post at the beginning of August with some update on the food portion of the plan as well as other things I might add. Discouraging but not disallowing eating out will keep me from just eating out instead of cooking the food I have at home. I’m hoping that the $20/week things will keep me from spending too much money and burning through the stuff I have at home. I doubt that I’ll have used up all the food in a month, but this should be an interesting experience and a good first step in minimalism.

There’s a chance I might start working on the room filled with stuff before the month is out, but I think going one step at a time is the best bet.

P.S. I’m going to try to log my weight, general energy level, how I feel, and when I sleep while I’m doing this. I think it could be interesting to know if just eating what I have in the house leads to something interesting. I also want to know right away if it’s going to cause problems.

Photo Credit: hansel5569

It’s been a while since I wrote anything on this blog. I’m feel bad about that, and it’s a food example of my last year.

Warning: This is a meandering, unedited post.

I started this last year with the expectation of getting quite a bit done since I’d have more free time. I also tried to follow the ‘no goals’ philosophy of Leo Babauta (of zenhabits.net) more or less.

It turns out that doesn’t work out for me. I respect Leo and love most of his content, but the idea of living without goals is something that I can’t attain, at least not yet. I’m more of a goal-oriented person. This is a personal post, so it might be a little rambling. Sorry for that.

Project Spawner

Prior to the start of this attempted philosophy I had all kinds of ideas for projects that could bring either personal satisfaction or financial freedom. This was the process each of them eventually went through:

  1. Have the idea – typically this wasn’t some “Aha!” moment, but more of an interest
  2. Think about and research the idea – The interest would develop, and I’d get deeper into whatever it was
  3. Find the actual business or project concept – This is where the idea goes from “A person could do something.” to “I could do this specific thing.”
  4. Research other businesses and start a plan – Sometimes it’s a proper business plan, but usually I just write down some ideas in a google document or something.
  5. Plan for scaling and do other research – At this point it gets pretty theoretical. This is where the stalling starts.
  6. Figure out why it’s impossible – I’ll find some reason why I can’t do it. It could be some issue with scaling or the risk of lawsuits. They’re excuses.
  7. Stop

I don’t recommend this process. It’s incredibly frustrating. This process came from a good starting place – setting goals. I would try to figure out what the steps were to start and succeed on a project, but in doing so I would overwhelm myself.

Over the last year I’ve started a good number of projects. Most of them have failed, but I’ll get to that later. Here’s how my process changed since I’ve been moving along without goals:

  1. Have the idea
  2. Think about and research the idea
  3. Start on a project

Better? Probably not. Having not thought through much the projects usually had a very sketchy start and several of them didn’t make it past that point. Actually, here are some of the issues I’ve run across when doing little to no goal setting:

  • Projects were ill-defined
    • This seems obvious, but without goals my projects didn’t have a clear direction or end point.
  • Projects did not end
    • Without goals you can’t tell when most projects are finished. That meant that most of my projects were unending. Even if the project was a little successful, I would end up abandoning it since I never had a true feeling of progress.
  • There was no reason to finish a project
    • Less of a reason, anyways. I didn’t have as much of that pang of guilt when abandoning a project and starting another one. As a general rule, I don’t think guilt is productive, but it can keep you going in limited quantities.
  • I didn’t consider transitioning or pivoting
    • A handful of projects could have worked if I had pivoted to something similar to what I was doing. Some others died when they needed to transition from starting to maintenance. These probably could have been handled with better goal setting.

There are many more issues as well, but you get the idea.

The manifestation of all this is that the projects never finished. It’s one thing to start a project and abandon it because it isn’t working out or I realized my skills weren’t up to par. That’s not what was happening. I was abandoning projects because I lost focus and got distracted. This can happen to anybody on any project, I know, but it happened to literally every project I started during this time frame.

It boils down to something I already knew, but had to re-learn:

Organization is Important!

Of course it’s important. Though, to be fair, organization and goals are not synonymous. The issue I had was that in doing away with goals, I did away with much of my organization. I had thrown the baby out with the bathwater. Knowing this, I could probably figure out how to be more organized without having goals, but I think that may be beyond me right now. At this point in my life, it seems that organization and goals are tied together in a way I can’t separate. Goals may create unnecessary stress for some people, but now I believe that this stress is necessary for productivity in some people (myself, for example).

Low Stress

One of the positive effects of going without goals for a while was that I had very low stress. There are two basic kinds of stress I had when I had been very goal-centric. I’d get stressed out by the things I didn’t know about – the unknowns of all of my projects – and I’d get stressed out by the sheer size and weight of the things I knew.

Oddly enough, getting rid of a list of tasks removed both of these. It turned out that when everything is unknown there’s no need to stress out about it. Part of the stress of unknowns is caused by having to have everything in a nice box. “Oh this task? That will take somewhere between 3 and 5 hours.” The problem comes up when I would have a task on the list and think “I have no idea how long this will take. Should this be its own project, or will it take less than a day?”

Those questions simply went away.

Moving on From Here

I know that I’ll keep coming back to this year of experience time and time again. Due to the low perceived cost of entry, I went into a number of different things that I never would have attempted otherwise. I broke free of some of the tunnel vision that had been keeping me from seeing opportunities. On the other hand, I ended up a long list of half-done projects. Most of them not worth finishing at this point.

I’ve learned two key things from this little exercise:

  1. I need goals
  2. When starting a project, goals are counterproductive.

Let me explain that second one a little. Goals in and of themselves aren’t counterproductive, but thinking about them too much too far in advance can be. I’ll explain.

When I first have an idea at t_0 I’m super motivated to get something done. As time goes on though, I lose that initial burst of motivation and there’s nothing else to keep me going. Then I hit t_1 and end up either distracting myself or giving up. This is not a good way to finish a project.

nogoalssketch

Let’s say I add some goals. I didn’t label things as well on this graph, but bear with me. Basically, all of the time up until the ‘effect of goals’ starts moving up is the time I spend figuring out the goals of the project. Typically this will be before I get going on the actual project. As you can see, it drags down my productivity and wastes the best time at the beginning. However, event if the goals are well formed and I follow through I often end up with something like this sketch:

withgoalssketch

You’ll notice it goes below the cutoff. That means the project dies. I give up, throw it away, or move on. I think this can be fixed by not starting on the goals until after I’ve dug in on the project. The reason that the effect of goals starts out negative is that all that time at the beginning that I spend working on goals puts a damper on my enthusiasm for the project. It also shows me the true scale of what I’m trying to accomplish. I’m just going to go out on a limb here and say that knowing the true scale of something is probably a bad thing most of the time. The project seems so immense that I think “there’s no way I could do that” even though I have done things that size or larger.

I think, and I want to test this, that going in with a general idea of what I’m doing right away, then setting small goals as I go in the initial steps of a project can get me farther than trying to come up with the 5-year and 10-year plans for a project from the beginning.

As an arbitrary cutoff, I’m going to say that I won’t plan out any project more than a month. I’ll try to keep this up to date as I go, but let me put down an example.

I want to write a nonfiction book. I have a few ideas on a specific topic and approach, but I haven’t nailed it down yet. Now, I know that writing a book that’s worth something is more than a one month project. That’s fine. However, my goals will all be things I can accomplish within one month. I know I won’t be done with the book by then, but it doesn’t matter. It’ll be done when it’s done.

Although I could dwell on the time I spent and how few of my projects came to completion, I’m going to instead learn whatever I can from the time and try to figure out how to apply it to the future. Who knows, maybe I can even write a book on goals at some point, from the perspective of someone who let go of them for a while without success.

I know this post was meandering and disjointed. I’ll do better on the next one. In fact, I plan to write about my experience trying to write and publish a nonfiction book right here on this blog. This will be my first big project with no long term goals, only mid and short term.

Because who can predict what life will be like 5 years from now.

 

I was working on reverse engineering a format for a side project (which I may post about later), and I figured I’d write about the basic process here. I was talking to some coworkers about my side project, and I got blank stares once I got to this part.

To demonstrate the technique, I’m going to use a nice, known format that I won’t get in trouble for picking apart. Applying this to closed format is an exercise left for the reader 😉
So let’s take a look at an h.264 encoded MP4.

These are both widely documented, and if you’re familiar with either of these, you’re unlikely to find anything new here. The point is to demonstrate my method of reverse engineering a format.

Let’s start out by assuming we don’t have a specification or any documentation on the format, it’s a black box.

I worked with three video files.
Video A – A short video
Video B – The first half of video A
Video C – A long, unrelated video

I picked these to have something similar to a known plaintext attack. Here we have two videos with the same initial content and different lengths, and another video. This gives us some basic separation between what is affected by file length and what is affected by file content.

OK, now we need to get these files open in some way where we can see the binary representation. I’ve been using Notepad++ with the Hex Editor plugin, but you can use whatever you want.

Typically (but not always) the first few bytes of a file is essentially a signature for the file type, it may also contain an internal type. This is where having two similar files is helpful. Here is the beginning of one of my files:

00 00 00 18 66 74 79 70 6d 70 34 32 00 00 00 00 ....ftypmp42....
6d 70 34 32 6d 70 34 31 00 00 12 02 6d 6f 6f 76 mp42mp41....moov
00 00 00 6c 6d 76 68 64 00 00 00 00 cb f3 0d b9 ...lmvhd....Ëó.¹
cb f3 0d b9 00 01 5f 90 00 09 37 2f 00 01 00 00 Ëó.¹.._...7/....

The left section is the hexadecimal representation, the right is the ASCII encoded representation. Anything displaying as a dot doesn’t have an ASCII representation.

From even this, it’s clear that this is a big-endian 32-bit format. What that means is that data is going to be stored in groups of 4 of these byte-pairs (or eight bytes if your editor doesn’t use pairs). We know it’s big-endian since it’s stored as “mp42” and not “24pm”.

You can also see that the string “mp42” is there twice, and “mp41” is there once. This looks like a signature (or header) to me.

Here is the data from movie B’s first 64 bytes, with differences highlighted.

00 00 00 18 66 74 79 70 6d 70 34 32 00 00 00 00 
6d 70 34 32 6d 70 34 31 00 00 0c 96 6d 6f 6f 76
00 00 00 6c 6d 76 68 64 00 00 00 00 cb f3 0d df 
cb f3 0d df 00 01 5f 90 00 05 63 98 00 01 00 00

Now we know where the signature ends. Now, remember, the data is stored in 32 bit pieces (let’s call them integers, or ints), so for video A the first data gets broken down like this:

00 00 00 18  ....
66 74 79 70  ftyp
6d 70 34 32  mp42
00 00 00 00  ....
6d 70 34 32  mp42
6d 70 34 31  mp41
00 00 12 02  ....

Based on this, we can assume that the last the last int is _not_ a part of the signature. Assuming we know nothing else about the format, we can assume this signature will tell a decoder or player what format they can expect, and perhaps a fallback – the mp41.

Now onto the next section. Once you start to see any kind of variable data, you’ll want to start by trying to determine the way the data is laid out. Basically, we want to figure out how the computer answers the question of “What is this?” for any particular piece of data.

I’ve found that most (not all) formats have their data stored in a more generic data structure. That way over time they can add things without necessarily breaking older readers/players. Because of that, the first thing I look for to determine the general structure of a file is pointers, offsets, and lengths.

  • Pointer – An address referring to a specific place in a file
  • Offset – Similar to a pointer, but relative to the current location in the file
  • Length – Opposite of an offset, it’s the offset minus current location
My rule of thumb is this:
If it’s relatively small, it’s likely a length
If not, check that address in the file and see if it looks significant, if it does it’s probably a pointer
If it doesn’t, add the current offset and check there, if it looks significant it’s probably an offset
If it doesn’t, this is likely some other variable, unrelated to the structure of the file.
In our example above, the next two ints are:
00 00 12 02   ....
6d 6f 6f 76   moov

Since we’re at the beginning of a file, I’ll check the location 1202. (Again, if this was little-endian, it would be location 02 12 00 00). Here’s what I have around the location 1202 in my file:

000011e0  00 00 00 3a 75 64 74 61 00 00 00 17 a9 54 49 4d  ...:udta....©TIM
000011f0  00 0b 00 00 30 30 3a 30 30 3a 30 30 3a 30 30 00  ....00:00:00:00.
00001200  00 00 0e a9 54 53 43 00 02 00 00 33 30 00 00 00  ...©TSC....30...
00001210  0d a9 54 53 5a 00 01 00 00 31 00 00 1e 13 75 75  .©TSZ....1....uu
00001220  69 64 be 7a cf cb 97 a9 42 e8 9c 71 99 94 91 e3  id¾zÏË-©Bèœq™"`a

It looks like we’re in the middle of a section, and the location is even in the middle of an int, so it seems unlikely this is a pointer. Maybe a length then? Let’s check what’s at 00001202 + 18 = 0000121a:

00001210  0d a9 54 53 5a 00 01 00 00 31 00 00 1e 13 75 75  .©TSZ....1....uu
00001220  69 64 be 7a cf cb 97 a9 42 e8 9c 71 99 94 91 e3  id¾zÏË-©Bèœq™"`a

That seems to make more sense. So 00 00 12 02 is probably a length. Let’s look at the next int: 6d 6f 6f 76 ascii: moov. This has to be a type, it’s a string and the idea of a type called “moov” in a video makes sense to me.

Woo! We have a section header. The next 12 02 bytes are part of the “moov” section. Now, if we go to 00 00 12 1a, we can see that this is another [length][type] pair. The next type being “uuid” Just to see if the theory holds up, let’s check out 0000121a + 00001e13 = 0000302D:

00003010  20 20 20 20 20 20 20 20 20 0a 3c 3f 78 70 61 63            .<?xpac
00003020  6b 65 74 20 65 6e 64 3d 22 77 22 3f 3e 00 00 00   ket end="w"?>...
00003030  01 6d 64 61 74 00 00 00 00 00 09 f6 85 00 00 03   .mdat......ö…...

This looks similar. The only difference is that 00 00 00 01 cannot be a length. Scrolling down a bit shows a wildly different looking type of data, so I think it’s safe to assume “mdat” means something like “movie data” meaning what follows is our movie data. Since mp4s can be streamed, it’s safe to assume that this is our h264 encoded video and nothing follows that.

So right now, from what we can see, the structure based on video A looks like this:

|---------------------------|
| Signature (outer wrapper) |
| |----------------         |
| | moov section  |         |
| |----------------         |
| |----------------         |
| | uuid section  |         |
| |----------------         |
| |---------------|         |
| | mdat section  |         |
| | |------------||         |
| | | video data ||         |
| | |------------||         |
| |---------------|         |
|---------------------------|

A quick check of the other two files shows the same superstructure.

Hmm, OK, so let’s dig deeper into those moov and uuid sections and see what we can come up with.

moov section

The next 16 bytes seem easy enough to figure out:

00 00 00 6c 6d 76 68 64  ...lmvhd

OK, we have a section called “mvhd” of length 6c. Simple enough, let’s go forward 6c:

00 00 06 c0 74 72 61 6b  ...Àtrak

A section called “trak” of length 6c0. Continuing on 6c0, we find:

00 00 05 28 74 72 61 6b  ...(trak

That’s interesting, another “trak” section. That must mean that not all of these are IDs, but some (or all) of them might just be descriptors that don’t need to be unique. Let’s keep going another 528:

00 00 00 3a 75 64 74 61  ...:udta

A “udta” section. Another 3a bytes later, we’re back here:

00 00 1e 0c 75 75 69 64  ....uuid

So that’s our “moov” section. Looking at the other two files shows the exact same format. So, from what we can guess so far, our format looks like this:

|---------------------|
| moov section header |
| |--------------|    |
| | mvhd section |    |
| |--------------|    |
| |--------------|    |
| | trak section |    |
| |--------------|    |
| |--------------|    |
| | trak section |    |
| |--------------|    |
| |--------------|    |
| | udta section |    |
| |--------------|    |
|---------------------|

We need to go deeper. Let’s go all the way down each subsection of the moov section.

At a glance, it doesn’t look like the mvhd section contains any more structural information, so let’s move on for now.

The first trak section is more interesting. The first 16 bytes in the trak section look like this:

00 00 00 5c 74 6b 68 64  ...\tkhd

Simple enough. Again, the tkhd section doesn’t appear to have any structural information in it. So lets move another 5c bytes to this:

00 00 00 24 65 64 74 73  ...$edts

The next 16 bytes look like another subsection, so let’s go down another level:

00 00 00 1c 65 6c 73 74  ....elst

Which appears to be the only section in edts, since 1c + 8 = 24. Moving on we have this

00 00 09 5c 6d 64 69 61  ...\mdia

Again, this clearly has some subsections, the next 8 bytes are:

00 00 00 20 6d 64 68 64  ... mdhd

mdhd doesn’t look like it has subsections, 20 bytes on we have:

00 00 00 44 68 64 6c 72  ...Dhdlr

This hdlr section actually has some human readable text! It does not, however, have any subsections. 44 bytes later:

00 00 00 14 76 6d 68 64  ....vmhd

No subsections, Next section is:

00 00 00 24 64 69 6e 66  ...$dinf

This one subsection:

00 00 00 1c 64 72 65 66  ....dref

Looks like like this one has a subsection:

00 00 00 0c 75 72 6c 20  ....url

Kind of a strange name, url[space], but it fits our expectation of a section header. By now you might be losing track of where we are, currently we’re still in moov->mdia

At this point, this post is getting far too long and repetitive, so I’ll just summarize the rest of this moov section. Using the same method we’ve been using so far, we find the moov section has the following format:

|----------------------------------|
| moov header                      |
| |---------------|                |
| | mvhd section  |                |
| |---------------|                |
| |------------------------------| |
| | trak section                 | |
| | |------------------|         | |
| | | tkhd section     |         | |
| | |------------------|         | |
| | |------------------|         | |
| | | edts section     |         | |
| | | |--------------| |         | |
| | | | elst section | |         | |
| | | |--------------| |         | |
| | |--------------------------| | |
| | | mdia section             | | |
| | | |--------------|         | | |
| | | | mdhd section |         | | |
| | | |--------------|         | | |
| | | |--------------|         | | |
| | | | hdlr section |         | | |
| | | |--------------|         | | |
| | | |--------------|         | | |
| | | | vmhd section |         | | |
| | | |--------------|         | | |
| | | |----------------------| | | |
| | | | dinf section         | | | |
| | | | |------------------| | | | |
| | | | | dref section     | | | | |
| | | | | |--------------| | | | | |
| | | | | | url section  | | | | | |
| | | | | |--------------| | | | | |
| | | | |------------------| | | | |
| | | |----------------------| | | |
| | | |----------------------| | | |
| | | | stbl section         | | | |
| | | | |------------------| | | | |
| | | | | stsd section     | | | | |
| | | | | |--------------| | | | | |
| | | | | | avc1 section | | | | | |
| | | | | |--------------| | | | | |
| | | | |------------------| | | | |
| | | | |------------------| | | | |
| | | | | stss section     | | | | |
| | | | |------------------| | | | |
| | | | |------------------| | | | |
| | | | | ctts section     | | | | |
| | | | |------------------| | | | |
| | | | |------------------| | | | |
| | | | | stss section     | | | | |
| | | | |------------------| | | | |
| | | | |------------------| | | | |
| | | | | sdtp section     | | | | |
| | | | |------------------| | | | |
| | | | |------------------| | | | |
| | | | | stsc section     | | | | |
| | | | |------------------| | | | |
| | | | |------------------| | | | |
| | | | | stsz section     | | | | |
| | | | |------------------| | | | |
| | | | |------------------| | | | |
| | | | | stco section     | | | | |
| | | | |------------------| | | | |
| | | |----------------------| | | |
| | |--------------------------| | |
| |------------------------------| |
| |------------------------------| |
| | trak section                 | |
| | |------------------|         | |
| | | tkhd section     |         | |
| | |------------------|         | |
| | |------------------|         | |
| | | edts section     |         | |
| | | |--------------| |         | |
| | | | elst section | |         | |
| | | |--------------| |         | |
| | |--------------------------| | |
| | | mdia section             | | |
| | | |--------------|         | | |
| | | | mdhd section |         | | |
| | | |--------------|         | | |
| | | |--------------|         | | |
| | | | hdlr section |         | | |
| | | | |--------------|       | | |
| | | | | soun section |       | | |
| | | |----------------------| | | |
| | | | minf section         | | | |
| | | | |------------------| | | | |
| | | | | smhd             | | | | |
| | | | |------------------| | | | |
| | | | |------------------| | | | |
| | | | | dinf             | | | | |
| | | | | |--------------| | | | | |
| | | | | | dref         | | | | | |
| | | | | | |----------| | | | | | |
| | | | | | | url      | | | | | | |
| | | | | | |----------| | | | | | |
| | | | |------------------| | | | |
| | | | |------------------| | | | |
| | | | | stbl             | | | | |
| | | | | |--------------| | | | | |
| | | | | | stsd         | | | | | |
| | | | | | |----------| | | | | | |
| | | | | | | mp4a     | | | | | | |
| | | | | | |----------| | | | | | |
| | | | | |--------------| | | | | |
| | | | | |--------------| | | | | |
| | | | | | stts         | | | | | |
| | | | | |--------------| | | | | |
| | | | | |--------------| | | | | |
| | | | | | stsc         | | | | | |
| | | | | |--------------| | | | | |
| | | | | |--------------| | | | | |
| | | | | | stsz         | | | | | |
| | | | | |--------------| | | | | |
| | | | | |--------------| | | | | |
| | | | | | stco         | | | | | |
| | | | | |--------------| | | | | |
| | | | |------------------| | | | |
| | | |----------------------| | | |
| | |--------------------------| | |
| |------------------------------| |
| ---------------------------------|
| |----------|                     |
| | udta     |                     |
| |----------|                     |
|----------------------------------|

Ok, that took a while. It’s fairly clear to me that some of these could change based on the specifics of what audio and video codecs you’ve used to encode.

The uuid and mdat sections do not contain any subsections. That is, the h264 blob inside of our mdat is still a black box, but we’ll get to that later. For now, I think this post has explained the basics of the process in reverse engineering a file format. Not all of them will be this simple, but most of them are easy enough to figure out.

Here are a couple of other basic ways I’ve seen data structured in a file:

  • An xml file, with none of this fancy pointer stuff. These are the easiest to figure out, since they are self-describing
  • A pointer to a directory which has a set of fixed-size entries containing the headers instead of a forward-read style of header
    • See the DOOM .wad format, which is widely documented
    • This is better when you need random access to some part of the file, instead of needing to read the file serially, like a video

I’m planning on writing up the h264 part of the file as well at some point. But I think we’re done for now. Due to what I know about mp4 since it isn’t a black box, I know that there can be more sections and subsections. The point wasn’t to show anything about the mp4 format specifically, but to show how easy it can be to piece together the structure of a file with nothing more than the file.

Have fun tearing apart files!

So, the personal development stuff is still interesting to me, but probably not interesting enough to write about.

Or at least write about consistently.

So instead, I’m going to start using this space to write about whatever I’m doing at the time. I might post updates when I upload a podcast, but mostly I think I’m going to write about whatever is interesting me. Which means likely code and programming.

So yeah, expect that. 🙂

Hey Internet.

Looks like extreme food budgeting during a month were I have to move may have been a bad idea. I couldn’t use my kitchen to any real extent for about two weeks, and as I’m sure you all know, pre-prepared and frozen food while they don’t have to be expensive, aren’t super cheap. I also lost track of my exact costs. I have a good estimate and I’ll do a proper postmortem later. I might even try this specific challenge some month in the future again.

November Challenge

NaNoWriMo

Probably not too many updates on this one, and no I won’t share what I’ve written. It’s sure to be terrible (as all first drafts are), and I haven’t written fiction since high school. This is more of a personal exercise than anything. If it’s amazing, I’ll keep going with it. It probably won’t be amazing, I’m not expecting it to be.

By the way, I started the novel with no plot, setting, or characters in mind. This should be interesting 🙂

I’m almost a third of the way through my October Challenge, and I’ve so far been fairly successful. Here are the basic numbers:

Amount spent: $25.82
Budget so far: $29.03
Difference: -$3.21
Difference in Days: –0.995

The difference in days is assuming that each day is worth $100/31 days, which is about $3.2258 per day. I’m fairly excited that I’ve been able to keep it under budget.

Reflections so far

My goal for the first couple of weeks here has been to go through all of the food in my refrigerator and freezer before I move on the 15th. At this point it’s looking like I’ll have most of it cleared out in time. As a result, I haven’t hit my reserve of rice too hard yet. I expect to start draining that after the move.

Probably the only completely unnecessary thing that I’ve purchased so far this month was a pizza. I had fast food on a day that I was on the road. I’m OK with that. I’ve been getting a burger for lunch at a bar with coworkers on Mondays, I think tomorrow will be the last time I get to do that. The soda so far hasn’t been excessive. Well, as least it hasn’t been as excessive my typical amount. I have been fairly successful with switching to tea at home.

Protip for anyone planning on switching from soda to tea: Green tea has somewhere between ½ and ⅓ the caffeine as most sodas. If you’re as addicted as I probably am, you’ll want to back off a bit at a time or have some other caffeine supplement for a while. Otherwise you’ll have extreme sleepiness and possibly headaches. I speak from experience.

Since I have a lot of flour and such around I made something resembling Indian Naan, but with a few changes to the ingredients depending on what I had on hand. I fried it in a skillet. The results were… edible. Not great by any means, but edible. I might take another stab at it, but probably not until next week. Also, probably not without milk, that seems to have been a mistake.

All in all, I’ve made it so far at or under budget. It’s not playing out exactly how I was expecting at first, but I’m happy with the experiment so far. I think it’s still too early to do any kind of analysis. I expect the last week to be the most interesting.

Log

Date Item Amount
10/3/2011 Dbl cheeseburger w/fries @ a local bar $6
10/6/2011 6 pk diet Dr. Pepper 24 oz bottles $4.09
10/6/2011 Pizza $6.15
10/7/2011 Burger King $4.21
10/7/2011 12 pk Pepsi Max 12 oz cans $5.37

Steve Jobs died yesterday.

I’m not going to write about his life or achievements. I’m not a person who can write an appropriate tribute. People more familiar with him have written words more eloquent than I could muster. Instead, I’m writing about something he said that affected me.

I’ve owned Apple products. My first computer was a Mac, and my first smartphone was an iPhone. The company and products Steve Jobs built have had an impact on my life.

This got me thinking. Reflecting.

What Are You Doing Right Now?

In his speech to Stanford in 2005, Steve Jobs talks about death. A somewhat morbid topic for a commencement address, but relevant. The quote that resonates with me was “Since then, for the past 33 years, I’ve looked in the mirror every morning and asked myself, ‘If today were the last day in my life would I want to do what I’m about to do today?’”.

Just let that sink in for a minute.

He goes on to say “And whenever the answer is ‘no’ for too many days in a row, I know I need to change something.”

Do you need to change something? I certainly do.

This topic has kept coming up for me, over and over again. Every time it stings a bit, and Jobs’ way of saying it is so grounded it stung hard. Let me share with you what I’ve learned on the topic.

How Do You Change?

For most of us, we can’t just quit whatever it is we’re doing today and start doing the thing we love or get the most satisfaction out of tomorrow.

We can, however, take steps towards that goal.

The first is to determine what’s important to you. It could be family, it could be religion, it could be your work, or anything else. Sometimes we all lose sight of what is important to us. Just take a while to sort it out. It may take a few minutes or a few days, either is OK.

Also, even if your job is just something that pays the bills I think you should consider it. You spend most of your week at your job, you should be doing something that you love.

Next, figure out what you can do tomorrow. If you want to change careers or jobs, your first step is research. Don’t quit before you have fairly concrete plans. If possible, line up your new training/job before you quit. If you need to change locations or move closer to family, look for real estate or rentals. Take a weekend trip and see if the place is what you think it is. If you’re going to need seed money, start aggressively saving or looking for investors.

In any case, it’s likely that you can do some research or planning right away. Don’t wait any longer.

Lay Out a Plan

You’re not going to know every detail. Life doesn’t work that way. Planning will give you a better idea of the potential options and pitfalls. You won’t be able to write a step by step guide. You should, however, be able to determine the first step. Put a date on it.

Take a few evenings and think through the pitfalls you might encounter and the worst outcomes you can imagine. Try to think of how you can avoid those pitfalls and outcomes. You won’t think of them all, that’s fine. You need to build the confidence and knowledge base to handle whatever life throws at you.

Control Your Fear and Live

When it comes time to take some of the first steps you’ve determined in your plan, take them. You will probably be afraid. That’s OK, just don’t let that fear stop you from trying to improve your life. If you’re not already on track, you will never achieve your life goals without changing anything.

When you start to fear the process leading towards your desired outcome, just remember that we all die eventually. None of us are getting younger. Waiting a few years will only make it harder and less likely that you’ll ever get there.

Life is too short to wait to live.

I’m in the middle of moving halfway across the state. I’m excited to get into my new place, but I’m not excited about the associated bills. To combat the costs, the first thing I did was stop buying things like video games, electronics, and so on. It seemed like there was more room to cut, so I took a step back and analyzed my food costs.

They seemed high.

For example, some quick accounting places my food costs last month (September 2011) around $450. Doing a little digging, it looks like the USDA averages in 2011 for a male 19 – 50 are $176 – $347. Again, my food costs are much higher than they should be.

I’ll be changing that, at least for a month.

The Reasons

Save Money. This is easily the simplest and easiest to understand reason. Dropping my food costs by a few hundred bucks a month would give me a little money to spend on something else (like moving expenses) or save for a rainy day.

Use What I Have. I’ve ended up with a lot of canned, frozen, and otherwise preserved foods that I haven’t been using. By severely reducing my food budget, possibly below a sustainable level, I’m forced to use up some of the food I’ve had just sitting around wasting space. This will help me clear up space and re-evaluate what canned and frozen foods I should keep around.

Gain some perspective. Aside from a few times in college, I’ve never really been in a spot where I couldn’t afford food. And honestly, those times in college weren’t all that bad. I’m hoping that by limiting my food budget I’ll gain a better perspective on the actual cost of the food I’ve been buying. Most of these basic facts are easily enough found online, but you don’t truly understand these things until you’ve lived them.

Minimalism. I’ve always been interested in minimalism, but I’ve never had the motivation to actually implement minimalism into my life. Just looking around the room I’m in, I see shelves packed with clutter. Moving is highlighting just how much clutter I have in my life. Perhaps this will give me a first step towards actually eliminating unnecessary things.

The Goal

Spend less than $100 on food in the month of October.

The Rules

I’m laying out the rules here more or less for myself. I’m not recommending this to anyone, this is more of an experiment. I might fail, and it might not be remotely healthy.

What Counts: Groceries, Fast Food, Restaurants, Soda, Vending Machines, etc. All food purchases count towards the budget.
One exception: In a couple of weeks some family is helping me move my bigger stuff (furniture and so on), and I’ll be buying them lunch. I’m counting this as a moving expense.

What Doesn’t Count: Food that I already have (since I already paid for it), and food that is given to me for whatever reason does not count (since I didn’t have to pay for it).

Since I’m doing this and making the rules, I’m not worried about gaming the system. There are probably holes in it somewhere, but since I’m not a hostile player, I feel confident in these simple rules. Oh, and no, I didn’t just stock up on food before October started.

The Plan

Last month, I spent somewhere in the ballpark of $100 on fast food and restaurants. First step is to cut that down or out. Simply, bring lunch to work.

The surprisingly large expense is soda. I spend probably $4 – $6 a day on soda. That’s about $120 – $180 a month. I’m planning on cutting that down as much as possible. First, I’m switch to tea at home and bringing my Black Zojirushi 17-Ounce Mug in filled with tea to cover the mornings. It’s a challenge to make good tea at work, so I’ll just go with water in the afternoons unless I’m in desperate need of caffeine.

Aside from those, my basic plan is to start by seeing how long I can go without buying any food at all. Then, once that’s no longer an option, try to be careful to use coupons and go with inexpensive options.

Conclusion

That’s more or less the plan. I’ll do a post every weekend on how this project is going. I’ll include how much I spent in the last week and more or less what I’ve been eating. These will be to keep me honest, I won’t go into too much detail.

Some time in the first week or so of November I’ll write up a postmortem to summarize my thoughts. I’ll try to get some good, money-saving tips out of the experience and write those up as well.

I’m not a professional writer, I’m not a journalist, I’m not a social media expert, I’m just a regular working guy. That’s not to say I don’t plan to make the jump to Internet entreprenuer, but for now this is only going to be a weekends and evenings thing.

So why write? I’m not exaclty sure. I’ve found that even though I’m not a writer, I write. I write to sort out my thoughts, I write to design, and I write to break down complex actions, and I write to find places to start. So, I figure that I might as well clean it up a bit and put it on the Internet for whoever stumbles across it. Maybe it will help someone, even if I can’t write on a professional level.

I write because I’m compelled to write. I hope you find my writings useful, entertaining, or both.

I’m planning on writing about my passions, which consist mainly of producitivy, personal time management, organization, technology, and mathematics. Lesser passions of mine include video games, anime/manga/cartoons/comics, and a frightening number of nerd stereotypes. I’ll probably be writing about those later ones less, but you never know.

I expect to have an About Me page up in the relatively near future I’ve posted an About Me page, so I’ll keep those interesting facts on that page, instead of this post. If you’re from the future, Good News! it’s already up. Go find it and read it if you’re interested. You must be one of those types who either wants to see where this (hopefully Internet Famous by, um, now) person came from or you’re one of those weirdos who reads a blog from the beginning. 😉