Beats + Kinect = Win

UPDATE: Demo video and full report released and new downloads available. See here for more details or see below:

Finished up writing the full report with instructions, etc. Includes all relevant documents, source code modifications, photos, and of course a video of the actual demo ; )

Report: 2011-05-09 ESE350 Final Project – Report.pdf
Video: Beats, Advanced Rhythm Game – 1.5.5b Kinect PoC Demo [Android]
Blog: Dance With Your Hands

For installation instructions, download all the files in the “install/” folder and skip down to “5. b)” in the report.

Happy hacking!




For the final project of one of my classes, I modified Beats so it would take input from an XBOX Kinect. The result is a Kinect hooked up to a Beagleboard-xM running Android and playing Beats. It was all proof-of-concept of course as everything ran with extreme lag, but it was pretty cool nevertheless and fun! Here’s a pic of my lab partner mastering Tetris (Rock Version) on Beginner[1]:

More pictures are posted posted on the Beats facebook page here!

For more info, see the (currently still being updated) blog here. The modified, signed APK can be downloaded here. The patch file (for inspection-purposes only) can be found here. The modified ofsample demo used for calibration can be downloaded here. The (possible?) source for the demo can be found here. Make sure to recompile your Beagleboard’s Android kernel with ‘CONFIG_USB_DEVICEFS=y’ and add ‘mount usbfs none /proc/bus/usb -o devmode=0666′ to your init.rc file as per instructions found here.

Happy hacking!


Post your comments and feedback in the forum thread here!

Looking for translators

To deal with managing the various translations/localizations of Beats, I have uploaded the strings to crowdin, a collaborative translation tool that supports Android.

If you are interested in helping out with the translations, please visit that site and start translating! In particular, I am strongly looking for someone to help out with Japanese translations. Once translations are complete, I will release a quick Beats 1.5.5b update which will include some of the updates on the Changelog page (particularly a setting allowing you to explicitly change the game language).

As always, make sure to join the Google group to keep up with updates. Happy translating (and happy Easter to those who celebrate it)!


Post your comments and feedback in the forum thread here!


Yay, the new version of “Beats, Advanced Rhythm Game” has reached over 100,000 downloads (with over 1000 ratings averaging 4.5/5 stars) \o/ Technically this happened months ago under the old package name but I just didn’t notice back then xD (the current cumulative total downloads so far is actually well over 250k at the moment ^_^)

Due to heavy school work, I haven’t had much free time to work on Beats apart a few random things listed on the Changelog page. Rest assure, however, that all your feedback and support has not been forgotten! Depending on the amount of free time I have over the summer, Beats 1.6b may either be implementation of new features or a complete rewrite with a much more stable and flexible engine (e.g. less prone to lag/miss bugs, support for BPM changes, customizable skins, etc.). Of course, everything on the ToDo page will be on the agenda!

Til then, spread the rhythm and have fun with Beats!
Check out Beats’ Facebook page and connect with other Beats fans!
Keep up with progress and find the latest development guilds via the Google group!


Post your comments and feedback in the forum thread here!

New Download Songs Page with Search!

Instead of redirecting to the Download Songs forums, the new Songs page now includes a custom Google search engine which will search through the below sites:


Try it out on your phone by clicking the “DOWNLOAD SONGS” menu item in Beats. Note that I guarantee neither the quality nor compatibility of song packs found via the search engine except official releases in Beats Portable’s Download Songs forums.

If you have suggestions of other websites to add to the above list, please post in the forum thread linked below~

Have fun and happy hunting!


Post your comments and feedback in the forum thread here!

Beats 1.5.4b

Due to a busy school and job-finding schedule, I haven’t had much time to work on any of the stuff in my huge To-Do List. The Beats user-base, however, has proven to be very eager in seeing the progress of this project and many individuals have actually helped me with translating Beats into other languages. Turns out, for example, the top four countries in terms of user base are USA (34%), South Korea (22%), China (21%), and Japan (6%) despite having no official Android Market in either China or Korea and no translations for any of the latter three.

In addition to various minor fixes as listed in the Changelog, thanks to the following users for helping localize Beats:

  • Russian: burdik
  • Italian: disaster.ita
  • Dutch: XWing Ace
  • German: pluri
  • Chinese: arthur020
  • Finnish: Patrik Selin
  • Spanish: Glas
  • Korean: bunny1

Apart from localizations, I am currently working on integrating Immersion’s new MOTIV haptic feedback technology, meaning you will literally feel the rhythm of your music in Beats 1.6b (or at least feel more engaged in the gameplay with new vibration effects). If you are interested in following the development of Beats and/or helping with translating Beats into your native language (日本は、私はあなたで探しています!), please join the Google group. As well, make sure to Like the new Beats facebook page (also on the website’s side bar to the right)!

For those unable to access the Android Market or wish to install Beats manually, you can download the latest version via the revamped Download page (use the last link on the page).


Post your comments and feedback in the forum thread here!

Beats 1.5.2b – URGENT UPDATE!

Due to some Google policies, Beats has been re-released under a new package name (now ‘‘ – its the same app, just a different Market download).

(or manually download the .apk from the Release page)

Please uninstall previous versions of Beats (aka 1.5b or earlier) and install the new package (just search for “keripo”, “advanced rhythm game”, “stepmania”, “ddr”, or “osu”). The donate version has also been removed – please use the Donate page if you wish to express thanks and help keep development rolling!

Because the Android Market will treat 1.5.2b as a new app, all previous ratings and comments will have been wiped, so go ahead and 5-star the new Beats and don’t forget to tell your friends to update! If you’re interested in helping with development testing or would like to help with translations (looking for Korean/Japanese/Chinese translators specifically), check out the Google group!


Post your comments and feedback in the forum thread here!

Beats 1.5b

After tons of work into adding new features and optimizing to reduce/remove lag and increase framerate, Beats 1.5b is finally ready!

Beats 1.5b has TONS of new features, as described in the Changelog. Some of the most important features include:

  • osu! Mod and scroll up game modes
  • various new tapbox layouts
  • much, MUCH faster now and with less lag (even faster than 1.3b!)
  • lots of nifty shortcut buttons on the main menu
  • Russian, Dutch, and Italian translations (looking for translators, see here)

Not sure what this new “osu! Mod” game mode is all about? Check it out in the new gameplay video below:
Beats, Advanced Rhythm Game – 1.5b osu! Mod Gameplay [Android]

Over the next few weeks, Matt and I will be working on setting up a song posting bot for the forums (please contact me if you are familiar with phpbb3 forum scripting), so expect to see a lot of new song downloads from various song packs – I have tons of Vocaloid, VGMP, StepMix, and Otaku’s Dream song packs ready for posting. In addition, I’ve created a new Beats Portable Google group where I will be releasing periodic development builds for testing/feedback and translations. Already planned for the next major Beats 1.6b update is Scoreloop integration, .dwi support (yes, I slack), .osu support (if peppy responds), and possibly even a sneak peak into new haptic feedback technology!

To keep things short (i.e. to prevent myself from ranting), hope you all enjoy the major update!


Post your comments and feedback in the forum thread here!

osu! Mod – Work in Progress!

Follow the latest updates (including links to the latest development build) via the new Google group at

One thing I’ve noticed many of my users requesting/mentioning is support for osu! files or a separate osu! port/clone all together. I’m a big fan of rhythm-based music games (mostly StepMania, Beatmania IIDX, and DJ Max) but I actually haven’t tried much osu! style games (I tried Elite Beat Agents a few times on a friends’ DS but thats the extent of things). After downloading the latest version of osu! and trying out a few beatmaps, I convinced myself that modifying Beats some bit could definitely lead to a new Ouendan-style gameplay. Eventually, I could make a proper, full-fledged osu! clone – Beats was always meant to be more than just a StepMania clone after all!

I hacked around with Beats a bit and got quite a nice osu! mod implemented. Pretty much instead of having the notes scroll down as arrows in normal StepMania-style, the notes appear as tap-able circles in random locations on the screen. The circles fade in with a focusing ring around like in osu!. Tapped notes will turn into a explosion/shout-box image while a missed note will turn into a red X. Currently the co-ordinates of these taps are randomized but will actually follow actual patterns in the future (Matt is currently working on an arc-pattern generator). No slides/spinners/etc. yet (its just simple timed taps) but I may try to implement that later by hacking up what is used for StepMania-style holds.

Beats, Advanced Rhythm Game – 1.5a osu! Mod (Work in Progress!) [Android]

The structure/backbone is there and close to ready for an actual osu! clone – one step at a time though xD. Support for osu! beatmaps (.osz format) is planned, but probably not before the major Beats 1.5b release (for which I still have to stop slacking off on adding .dwi support). The major barrier would really be me not actually understanding what I’d be parsing – for that, I’d have to play osu! a bit more :D

For those who are curious and anxious to test things out on their own Android phone, you can grab the latest development build here:

Note that the development build uses a different signature, so you must first uninstall any official release before trying the dev build. The development build also includes a lot of new stuff/updates (it’s pretty much like 1.4.9/pre-1.5b), some of which aren’t 100% complete yet. When they are ready, however, you can look forward to an official Beats 1.5b release ; )

The current discussion thread on the osu! forums can be found here:


Post your comments and feedback in the forum thread here!

Tapbox Layouts and Scroll Up

One of the most requested features for Beats is different tapbox layouts (aka the area where you tap to hit falling notes). Ever since Beats 1.0a, notes always fall down from the top of the screen to the arrows at the bottom, with the tapboxes being invisible rectangles on top of the arrows. This is akin to the Beatmania/DJ Max “reverse” scroll where notes fall down and the same setup I’ve always used when I play StepMania (for which I usually play with 3.0x, Reverse, Notes+ and Dark modifiers). The majority of new users, however, are only familiar with DDR-style scrolling up and a D-pad like arrangement. And so after playing around with the code, I’ve added two new settings that will be available in the next major Beats release (1.5b): “Direction” (scroll direction of notes) and “Tapbox Layout” (arrangement of tapboxes).


The current placements are as followed (keep in mind that the grey boxes are just temporary placeholders that will be replaced with actual pad graphics once I get around to making some). The “Tapbox Overlap” and “Tapbox Placement” (previously called “Arrow Placement”) settings control the size and Y-axis shifts of these boxes.

D-Pad and Standard layouts w/ Standard/Up scroll

Reverse-V and Fullscreen layouts w/ Standard/Up scroll

D-Pad and Standard layouts w/ Reverse/Down scroll

Reverse-V and Fullscreen layouts w/ Reverse/Down scroll

These will be the new tapbox layouts you can expect to see in Beats 1.5b (which, as you can see in the FPS counters in the screenshots, is much faster and smoother).

Any other possible layouts you guys would like to see?


Post your comments and feedback in the forum thread here!

Rant on Optimizations -> Faster Beats 1.5b?

Warning! Long, random, technical rant ahead! Read at your own risk!
I may also misuse a few words incorrectly here and there – I apologize in advance for any incoherence within my rant


Modern Android

According to the latest data from Google, the majority of phones these days run Android 2.1 (Eclaire) or 2.2 (Froyo) and are most likely sporting mean, snappy 1GHz+ processors capable of even running advanced, graphically/arithmetically-intensive programs such as PlayStation emulators decently. With Android 3.0 (Honeycomb) not too far off in the distance and expected to require top-of-the-line 1GHz processors (or even dual-core processors), expect to be using your next generation smartphones for far more than just simple communication. If you look around today, you may occasionally spot old, cranky Android phones (like the Motorola Cliq, HTC G1, or even the original Motorola DROID) stumbling along with its 500MHz processors and incomplete multitouch touch-screen (or sometimes lack thereof). But those ancient Android 1.5/1.6 phones are easily eclipsed by the newer high end phones such as the Samsung Galaxy S, Droid X, and even Nexus One, all running 1GHz processors with complete multitouch support and capable of running processor-intensive apps such as Beats without a hitch (no, I haven’t actually done any real benchmark test comparisons as I do not own or have access to any phones other than my own, but these tests can give you a nice comparison point).


With these high performance wonder machines right around the corner, one might think it silly to even wonder, “why bother even trying to optimize things when its going to be running full speed anyway?” The answer from me is, “because I can.” Back when I worked on the iPodLinux project, it was all about pushing things to the limit. Sure, I didn’t have the coding knowledge to program things like dynarec in Ducky and Fellni’s iBoy project, I was always fascinated in making things run fast on my little first-generation iPod nano, strongly encouraged by the fact that my hDoom port easily ran at full speed and much faster than the initial iDoom port it was inspired by/based off. The ultimate flag was my igpSP port of Exophase’s popular Gameboy Advanced emulator. Despite initial skepticism of seeing a playable GBA port on the iPod, igpSP 0.9-2xb K7 (available only in SVN) ended up running most games at roughly 70% speed. So what does this all have to do with Beats? Nothing directly. I’m just ranting about why when it comes to making things run faster, aka making sure Beats runs smoothly and lag-free even on old phones that nobody really cares about, I try anyway and don’t give up easily.

Beats 1.5b?
With so many radical internal code changes between 1.3b and 1.4b, we decided to start taking a deeper look into optimizing Beats up. If you recall, Beats 1.0b was literally coded from scratch and tossed together during the 48hr PennApps 2010 hackathon where performance and code efficiency was dropped in favour for functionality. So after two SVN repos switches and approaching 300 revisions in the current repos, its time we sidetrack a bit away from the bells and whistles planned for Beats 1.5b and take a look at performance.

Summarizing the last week or so, we’ve made quite a few improvements in the current Beats code pertaining to performance. Neither Matt nor I can actually confirm that these improvements will have a real impact on the gameplay experience across the wide spectrum of Android-supporting devices – between the two of us, I have a fast Samsung Captivate that has always run Beats without issue ever since 1.0a while Matt doesn’t even own an Android device – he’s stuck with a slow, clunky emulator running on his Macbook. Nevertheless, the raw numbers are looking better and the FPS counter is going up. Here’s a few brief notes on what we’ve come across:


Traceview is Google’s nifty Android profiler. After playing around with it, I ran it on the two methods I thought were the most performance critical – the game’s onDraw method (which draws the arrows on the screen) and the notes’ update method (which updates the arrays of falling notes, missed notes, etc.). The results weren’t surprising, but very much reassuring on what needs to be optimized. For update(), the iterator loops seem to be used far more than they should – each call currently creates 8 queues that are reconstructed on every update (effectively every frame). Not only does the iterators loop through each note object multiple times, each queue creates new objects for each relevant note. The result is a huge mess of loops and (unnecessary) object creations that can definitely be optimized through many approaches.

Current plan involves experimenting with static arrays, the optimized System.arrayCopy where applicable, and running it all on a second thread with locking. Whatever approach we ultimately decide on, it will be more efficient than the current massive “bruteforce” mess. For onDraw, drawBitmap took the stage (as expected of course). What we did find surprising was the total time spent – 109ms in onDraw() vs 2ms in update(). Sure we can optimize away the backend however much we want, but whats the point when graphics accounts for 99% of the slowdown anyway? Current things in mind that we are considering looking into include current bitmap scaling, “dirty” redrawing, clipping used in the holds, static bitmap objects, etc. The possibility of just rewriting everything in OpenGL (and letting/hoping the library takes care of making sure everything is optimized) is still there, but with both Matt and I busy every day with our univ classes and studies, it probably won’t be happening any time soon. Besides, its always a lot more fun when you do things your own way from scratch ; )

Micro-Optimizations“Designing for Performance”
Some of the advice there made sense but I found much of it silly. “Avoid Internal Getters/Setters” – while I completely understand the logic and reasoning behind it all, I just find it silly why there isn’t any preprocessor that is smart enough to optimize out the overhead of method calling for single-line get/set methods (aka force inline everything or something along those lines). To be honest/fair, no, I haven’t actually looked too deeply into the Android build/run process, but the fact that the process of un-encapsulating all the data fields (lol-Java-OOP-concepts) and increasing the number of static methods has lead to slightly faster code makes me wonder why Google chose Java for embedded device programming. That said, however, I love Java and I love Google, so I’m not really complaining, just pondering on answers outside my scope of knowledge.

Background Redrawing“Speed up your Android UI”
This hack was god-sent. We already knew that background drawing was a major contributor to low framerate, but removal of it would mean no pretty picture to look at while you tap away furiously at your fragile phone. Neither of us suspected, however, that this entire time, the phone’s actual background was also being redrawn on each screen update, even if we were using an opaque background. I’m sure there’s plenty more graphical optimization tricks out there – we just have to keep looking!

Haptic Feedback Lag
This might be entirely a Samsung Galaxy S thing, but through a bit of profiling and experimentation, I discovered that a major source of lag actually came from the vibrate calls from taps (or note misses). One way or another, too many calls to Vibrator.vibrate() was somehow causing the phone to hiccup (blocking the UI thread somehow?) and especially problematic when the screen fills with notes and the user panics and reacts by just spam tapping all the arrows. Each tap calls the vibrate function, which in turn slows the phone down and makes the game even choppier, resulting in even more panic. A viscous cycle. Having added haptic feedback specifically such that the user is able to feel the game actually being responsive to his/her taps, I am very reluctant to remove the feature or turn it off by default.

Some quick Google searches of the web and the Android developer’s site/blog returned nothing and I doubt there even are many active/performance critical apps out there that even use the vibrate service, so I doubt I’ll be able to find out anything more than what I discover through experimentation. I tried moving the vibrate call to an AsyncTask, but it resulted in noticeably delayed vibrate responses that would easily lead to very distinct and prominently perceived lag (even though the actual game itself ran slightly smoother). Whether this was due to the AsyncTask overhead or the Vibrator service being slow, I don’t know but probably won’t look much into because the trade-off of performance over user experience just didn’t seem worth it. I was, however, able to spot my silly mistake of allowing for multiple vibrates per game cycle (i.e. request for a vibrate multiple times even when the arrows are being tapped at the exact same time/frame). Removing those redundant requests definitely did lessen the lag significantly, but apart from that, I don’t see much else there can be done without sacrificing responsive haptic feedback. Just got to optimize elsewhere I guess.

Garbage Collector
So there were a few reports of random lag spikes here and there that were clearly not associated with any particular event. So I played around with Beats the past few days and did indeed encounter a few instances of these random lag spikes. Only half a second in duration, they were not caused by any background data, any phone background process, or even a mass influx of notes in the game itself. In fact, they were seemingly 100% random and impossible to reliably replicate. After much thought into the possible cause of the problem, we thought of one (now blatantly obvious) culprit: the Garbage Collector! (no, I haven’t actually gotten around to finish watching the “Google I/O 2009 – Writing real-time games for Android” or “Google I/O 2010 – Writing real-time games for Android redux” videos but I plan to one day when my classes/schedule allows for it).

At the moment, I am highly suspecting the frequent calls to the GC are due to the largely inefficient iterator(s) described earlier and the numerous, unnecessary objects that are created. There’s two approaches to this: 1) explicitly call the GC when things aren’t laggy, or 2) reduce memory usage so that the GC doesn’t even need to get called at all during gameplay (where even a split second of lag could lead to a C-C-C-C-COMBO BREAKER). 1) Would work nice if we could predict at what times in the song is it alright to lag the phone and/or when there isn’t any upcoming note in the next many milliseconds. Unfortunately, such is very hard to predict (boo, theoretical textbook scheduling schemes don’t always work in the dynamic real world) and won’t work if the song you play happens to be spamtastic and full of non-stop notes. 2) of course, is the preferred method.

How would we reduce memory usage? For one, I am very much hoping that whatever approach we take for optimizing the iterators will also result in drastically lowered object creation and subsequently less GC calls. But even that isn’t a complete guarantee as what if its an old phone running Beats with limited RAM? Another alternate approach we were thinking of is the “buckets” approach. We pre-allocate a certain number of falling note objects and only allow for those to display on the screen. Once a note object is removed (either by being hit or missed), it returns to the pool of available objects for reuse. Pro of this approach would be that the GC will never need to be called as there is nothing to collect. Con of this approach is that we don’t know what is a good size pool (this would be hardware dependent wouldn’t it?). Even after we choose a good size, can there arise a situation (1.0x, jumps on, Challenge) where there is far more notes that need to be displayed on screen than the pool can hold? Hopefully the iterator optimizations will make interruptive, random GC calls an extremely rare event, but if not, the buckets approach I think is worth a try.


(for the stuff that I did try prior to typing out this long rant)
When I first heard about users’ wishes for smoother gameplay on their older phones, I decided to add a simple live FPS counter (and later also a cumulative one) just to get rough comparative numbers. On my Samsung Captivate, with the refresh rate set at 40Hz (i.e. up to 40 redraws every second, one invalidate call every 25ms, resulting in a cap of 40 FPS), I would consistently run at a comfortable 26-28 FPS during simple gameplay, maybe 24-26 FPS when the screen gets busy. Before the vibrate lag fix/tweak, framerate would drop down to 19-21 FPS if I spam tapped; after the fix, it only drops down 2 or 3 FPS. Meaning no more crazy lag when you panic and spam tap your phone to death (or at least less lag). The accumulated micro-optimizations that Google’s documentation recommended improved things a bit; while I couldn’t notice any graphical difference (at best I could say I saw my phone peak occasionally at 34 FPS, but that may have just been a placebo effect), Traceview said something improved, and something is better than nothing (I really need to implement some real benchtesting functionality for Beats one day…).

The most significant boost, however, came from the background hack. Average FPS went skyrocketing from 28-32 FPS to 38-42 FPS (I actually had to raise the refresh rate to 50Hz since it was being capped by the 40Hz’s limit). If I turned off the background image entirely and set the refresh rate to 60Hz, the game would run at an incredibly smooth 52-56 FPS, dipping down no further than 40-45 FPS at any given moment in time (as mentioned earlier, however, these actual numbers aren’t 100% accurate as I re-calculate the FPS every 10 frames based on time elapsed between the two calculations – I only use these numbers for comparative purposes). Of course, I can’t wait to try it out on another phone, but if it leads to a smooth 20+ FPS on even a Motorola Milestone, I’ll be happy enough in terms of optimization (of course the internal loops still need to be redone, but that would be for programming and design sake and not whatever small performance improvement results from it).


That’s it for now in terms of my rant.
tl:dr –> Beats 1.5b is still very much in the works, but when its ready, in addition to having a bunch of cool new features, it will run so fast and smooth on your old phone that you might have to worry about breaking your touch screen from playing too much Beats ; )


Post your comments and feedback in the forum thread here!

Go to Top